How do "decision models" like Jev return an answer with probabilities in milliseconds, and are those probabilities calibrated? I spent some time exploring this, with experiments on open Qwen2.5 1.5B and 7B models. I wrote about it all here (interactive): victordibia.com/explainers/j… TL;DR: decision models fit decision-style tasks (given some state, pick from a list), and they work in 4 steps: - Start with the decision: a state and a list of options. - Write it as a sentence to complete, ending with answers appended as options. - Score every option as that completion, all in parallel. (note: no text generation). - Normalise the scores and select the option with the highest score as the decision. 3 interesting results from experimetns on open models (not the Jev API): - Scoring was 7x to 54x faster than asking the model to generate the probabilities (what we might typically do with an autoregressive LLM) - Scoring was at least as accurate as generating a label. At 10 options, the two picked the same answer on 98.5% of messages (dataset used and other details in the post above). - Base models were close to calibrated; instruct models were overconfident. At 77 options the 7B instruct model was 95% confident on average and right 64% of the time. Temperature scaling can help (brought calibration error from 0.10 to 0.03), but that needs labelled examples from your own task. The natural next step here is can smaller models (e.g., 0.5b models) be finetuned to get closer to what 7b does and where do things fall apart? Next post. P.S. Like I mentioned earlier, I am experimenting with a format where my articles have an interactive explainer version, built on the experiments where possible and hosted on my website (see victordibia.com/explainers/j…). It's a lot more work, but I find it helps me understand, interrogate and teach concepts much better. References [1] victordibia.com/explainers/j… Interactive Jev Explainer