Decision Model vs LLM: Accuracy on Scoring and Classification, and When to Use Each

Short answer

Use a decision model when every possible answer can be written down in advance (yes or no, one of N labels, a level on a scale) and the call runs on every request or inline. Use a generative LLM when the output is text or a value nobody listed beforehand, or when a person needs to read the reasoning. Use plain code before either for anything that can be computed exactly.

The LLM is more accurate, but not by much. On Banking77 intent classification (3,080 messages, 77 intents), the Jev decision model was 81.0 % accurate against 84.4 % for Claude Opus 5, at about a thirteenth of the median latency and a twenty-second of the cost. On 17 quality-scoring tasks (170 held-out labelled items), an LLM judge had a mean absolute error of 0.089 on a 0 to 1 scale, Jev 1.13 0.104, and the OpenAI Decisions API and Cloudflare's open-weight Clef models 0.134 to 0.145. The decision models put pairs of items in the right order at least as often as the judge, partly because the judge's coarse 0.1 score steps produce ties. They answered 9 to 38 times faster and showed little or no variation between the two passes we ran. Which LLM you compare against moves the result more than any of these gaps.

Key facts

• Definition: A decision model takes a state (the input text) and typed questions and returns probabilities, with no generated text. A generative LLM returns tokens; as a classifier or judge, you parse a label or score out of what it writes.
• When to use: A decision model for closed questions on every request or inline: routing, classification, gating, scoring and checking another model's output. An LLM for prose, new values, open rubrics and explanations. Code for arithmetic, dates and counting.
• Limitations: About 3 points of accuracy, or 0.015 of mean error, behind the best LLM in these tests; no explanation; results move with the wording of the question; documented weak spots on counting, numbers, dates, multi-hop reasoning and injected instructions.
• Example: An insurance claims inbox where code checks dates and amounts, a decision model routes and flags injuries, an LLM drafts the reply, and the decision model blocks any draft that promises coverage.
• Jev 1.14: Released 2026-10-14; TypeSafe said
  would move to it a week later. The figures in this post come from
  and have not been re-run on 1.14.
• Last updated: 2026-11-07

Key takeaways

• The shape of the answer decides, not the difficulty of the task. A closed answer set points to a decision model; an open one to an LLM.
• On classification the frontier model wins by 3.3 points (95% interval 2.3 to 4.4), a real gap at n = 3,080. The two agree on 89.3 % of messages, and each is better on different intents: Opus led on 35, Jev on 15, and 27 were tied.
• On quality scoring the LLM judge had the lowest error on 9 of 17 task types and Jev on 6, and Jev was within 0.02 of the judge in five of six categories. The decision models' higher ordering accuracy is at least partly an effect of the judge's coarser output, not proof that they rank better.
• Variation between runs was low for the decision models: Clef and OpenAI Decisions gave identical answers across the two passes we ran, and Jev varied by an SD of 0.002, against 0.017 for the judge. Two passes show low observed variation, not determinism, and the wording of the question can still move results.
• Confidence ranks but is not a probability. Escalating Jev's answers below 0.90 confidence to the frontier model reached 84.0 % with Jev handling 76 % of traffic, an upper bound.
• In our tests the LLM chosen was the largest single variable. "LLM" without a model name, settings and a prompt design is not a result.

Definition

A decision model reads natural-language input and answers typed questions with probabilities instead of text:

A generative LLM produces tokens. As a classifier it writes one label from a list, ideally under a strict JSON schema with the labels as an enum. As a judge it writes a score against a rubric and usually a justification. Any confidence it reports is more generated text.

The models tested below:

• Jev 1.13 (TypeSafe), a hosted model trained for decisions, in early access since 2026-09-15.
• Clef 27B and Clef-flash 9B (Cloudflare, released 2026-10-01), on Workers AI, with weights under the Apache 2.0 license. Cloudflare says Clef follows Jev's request format, so switching is a change of endpoint and model name. We sent identical request bodies to both.
• OpenAI Decisions API (
  , model
  , public beta since 2026-10-06), a general model behind a decision-only interface with its own request schema.
• Claude Opus 5 as the frontier classifier and gpt-5.6-luna as the LLM judge.

When to use each

The choice is worth making explicitly when a call sits on a hot path: routing every ticket or agent step, classifying every record, gating a tool call or an outbound message, or verifying an LLM's output before it ships. If the call runs a few hundred times a day offline, pick whichever is more accurate on your own labels. If it runs on every request, the difference between a 0.2 second and a 2 second call, or between two cents and three dollars per thousand, decides the architecture.

Work through these questions in order for each judgment, and stop at the first "yes".

If you reach the end without a "yes" (a closed question, low volume, offline), either works: run both on your own labels, as described below, and keep the one that agrees with them more often.

Code first. TypeSafe lists counting, numeric comparison, date ordering and duration checks as things its model does not do reliably and recommends code for them. LLMs are not reliable at them either. Neither should be asked whether 14 March falls inside a policy period: extract the date and compare it in code.

Shape, not difficulty. "Is this message a prompt-injection attempt?" sounds hard, but its answer is yes or no. "Summarise what the customer wants" sounds easy, but its answer is open text. In a third-party test run on 2026-09-19, a decision model and a frontier general LLM both screened 40 of 40 messages for prompt injection correctly; on 60 support tickets they matched on intent (59 of 60 against 60 of 60) and escalation (60 of 60 against 59 of 60). Median latency was 0.194 s against 1.957 s, and cost $0.0248 against $2.88 per 1,000 tickets. These are small single-day samples: the shape of the tradeoff, not a leaderboard.

One small judgment per question. A decision model reads literally. "Does the claim involve an injury that happened during the policy period and exceeds the deductible" hides three judgments, two of them arithmetic. Ask "does the email say someone was physically hurt" and let code handle dates and amounts. Accuracy also falls as unrelated content grows, per TypeSafe, so send each question only the fields it needs.

Accuracy evidence

Two tests follow: classification against a frontier LLM, and quality scoring against an LLM judge. Treat every latency figure as indicative: a typical figure from one setup over the public internet, not a guarantee. Yours will move with region, payload size, questions per request, concurrency and provider load.

Classification: Banking77 against a frontier LLM

The published test (OpenRouter, run 2026-09-22) used the Banking77 test split (Casanueva et al., 2020): 3,080 customer-support messages, 40 for each of 77 intents, with a median length of nine words. Several intents are deliberately close, such as

The setup was fair on inputs:

• Same criteria. One line per intent, written from the label name only, without looking at test data. Both models got the identical list, and that list defines the task.
• Jev 1.13. One
  question per message, with the 77 descriptions as options.
• Claude Opus 5. Descriptions in the system message, temperature 0, reasoning off, a strict JSON schema with a 77-value enum, and prompt caching. The label list cost about 3,750 input tokens per request.
• Same client, same window. Eight concurrent requests from one machine in one fifteen-minute window; latency is the client-observed round trip.

Neither model returned an invalid response in 3,080 attempts. Every error was a wrong label, not a parse failure.

On the disagreements, Opus was right and Jev wrong on 175 messages, and the reverse on 72. A McNemar test on those counts (our computation from the published figures) gives a chi-square of about 43, so the gap is not noise. Both models sit well below the low-90s accuracy the original paper reports for encoders fine-tuned on the 10,003-example training split, the price of classifying from one-line descriptions.

Per-class errors tell you more than the headline:

• : Opus 80.0 %, Jev 52.5 %.

• : Jev 95.0 %, Opus 70.0 %. Opus tended to read stolen card details as an unrecognised payment.

• : both 0 of 40, mostly sent to
  . The intent covers the PIN arriving separately, which cannot be guessed from the label name. Excluding it, Jev reached 82.1 % and Opus 85.5 %.

The last is a criteria bug, not a model failure, and a confusion matrix on a development split catches it before the test set does.

Confidence ranks well, but is not a probability. TypeSafe defines a

The cascade. Accept Jev's label at or above a confidence threshold and send the rest to Opus:

At 0.90 the cascade gives up 0.4 points for about 3.5 times lower cost. The thresholds were read off the same items used to score accuracy, so this is the best case.

Quality scoring: 17 tasks against an LLM judge

The tasks are production LLM-as-a-judge evaluators, each with its own prompt, few-shot examples and a labelled calibration set never used to write or tune it. Labels are values from 0 to 1 for properties such as readability, politeness, helpfulness, safety for children or sentiment. We started from 23 evaluators and set six aside, for two unrelated reasons:

• Two had inconsistent labels. This is a reason about the labels themselves, and it is listed separately from the next four for that reason.
• Four were set aside because no model beat the label-average baseline. That decision used the measured results, so it is a selection on outcomes. Failing to beat the baseline does not prove the labels are bad. The tasks may simply be hard, or a poor fit for every model tested. Read the 17-task figures below as results on the tasks where at least one model did better than guessing the average, not as results on all our evaluators. Including the four would raise every model's error. Because no model did well on them, it says nothing in favour of either approach.

The remaining 17 task types fall into six categories.

Every model got the same evidence:

• LLM judge: gpt-5.6-luna running the evaluator's full prompt, including its few-shot examples.
• Decision models: one
  question per task, written from the evaluator's prompt by an author who never saw the test items, with the same few-shot examples attached to the levels they illustrate. The four task types without examples got none on either side. Each
  was divided by the top level number to match the labels' scale.
• Two passes per item, per model, to measure run-to-run variation.

We report mean absolute error (the average distance from the label; off by 0.1 beats off by 0.5) and pairwise ordering accuracy (the share of differently labelled pairs put in the right order).

Category values are unweighted means over task types; lower is better. Predicting each task's average label scored 0.296.

Jev tracks the judge closely everywhere except tone and rhetoric, where both Clef models beat it. The Clef models fall furthest behind on readability and on tasks that compare the text with a source or an instruction. OpenAI Decisions is close to the judge on sentiment and text transformation, furthest behind on tone, and tends to score good items too low.

Read the ordering column as partly an effect of output resolution. The judge writes scores in steps of 0.1, so it often gives two differently labelled items the same score, and a tie is not a correct ordering. The decision models return a probability-weighted score with many more possible values, so they rarely tie. Part of their lead on ordering therefore comes from the output format, not from better ranking. A fair test would let the judge write finer scores, or count ties as half right for every model. We have not run either, so treat the ordering result as unproven.

The judge's latency includes about one second of client start-up, while the decision models were called directly over HTTP; without that second the judge is still 8 to 32 times slower.

The judge's average run-to-run SD of 0.017 hides a tail: up to 0.028 over a category and 0.14 on a single item. The decision models showed little or none of that variation across our two passes. Two passes cannot show that a model is deterministic, only that it did not vary in this test.

Variation between runs is not the only source of instability. In earlier runs of the same tasks, averaged over two or three alternative wordings of each question with and without examples, Jev's error was 0.111 instead of 0.104 and Clef 27B's 0.126 instead of 0.145. Those runs changed the wording and the examples at the same time, so they cannot separate the two effects. They are also not comparable with the judge's SD, which measures something else: one is a change in average error across a task set, the other is per-item variation between identical runs. They do show that the question design can move a decision model's result, and that the size of the effect differs by model.

Few-shot examples do not transfer between decision models. Examples on the levels lowered error for Jev and OpenAI Decisions and raised it for both Clef models, which in the cases we inspected pulled scores toward the middle of the scale. Clef accepts the same request as Jev, but the same request does not mean the same behaviour.

The LLM you compare against is the largest variable

Each test uses one LLM in one configuration. In our readability-ordering experiment (2026-09-20, 8 held-out items, so indicative only), the same judge rubric scored 0.786 pairwise ordering accuracy on a gpt-4o-class model, 0.750 on gpt-5.4 and 0.964 on gpt-5.5, while Jev scored 0.929 to 1.000 depending on how the levels were worded. Against the gpt-4o-class model the decision model won clearly; against gpt-5.5 it landed above or below depending on wording, and on absolute error the gpt-5.5 judge beat most Jev designs. The spread between LLMs was larger than the gap between Jev and the best LLM.

Wording is the other large variable: rewording only the level descriptions moved Jev's held-out error from 0.038 to 0.079 in the same experiment, and

How to run this comparison on your own labels

A public benchmark tells you whether a comparison is worth running. Only your own labels tell you which model to deploy, and they turn "which model should judge this" into a measured number per question rather than a vendor claim.

1. Start from labels you already have. An existing LLM classifier or judge with a labelled set is the natural start: the labels define the target, the LLM is the baseline.
2. Sample enough items. At about 85 % accuracy, 400 items give a 95% interval of about plus or minus 3.5 points and 3,000 items about plus or minus 1.3. Score both models on the same items and test the paired difference (bootstrap or McNemar); at 400 items a 3-point gap is usually not separable from noise. Forty items per class at 80 % still leave about plus or minus 12 points, so read per-class results as a ranking of problem areas.
3. Split, and keep test items out of the question. Write descriptions, levels and thresholds on a development split; touch the test split once. An item used as a few-shot example, or looked at while writing a level, cannot be a test item.
4. Write the question once, version it, and give every model the same evidence. Build every request from one versioned file: the decision model's
   , the LLM's enum and system message, and the same examples on both sides or neither. Keep vendor details in the request-building code. Clef takes Jev's request unchanged (Workers AI wraps the answer in
   ). The OpenAI Decisions API needs a translation: the input is a string, questions are an array of objects with a
   , each level has a
   and a
   , and an answer of type
   must be handled, not read as a low score. A reworded question, or even one reworded option, is a new version.
5. Fix and record every configuration. LLM model and version, reasoning, temperature, schema, examples and caching. Pin the decision model:
   moved to
   a week after the 2026-10-14 release, so a client sending the alias got a different model with no code change. Store question and model versions with every result.
6. State the decision rule before scoring the test split, for example the largest paired accuracy gap or error margin you will accept. A rule written after seeing the numbers cannot fail.
7. Normalise the same way for every model and run each item at least twice. Divide each
   by the top level number, or better, value each level and take the expected value over the returned probabilities. Repeats separate real differences from sampling noise.
8. Report against a baseline, then look inside. For scoring: mean absolute error, pairwise ordering and the error of predicting the label average; a model that does not beat the average says nothing about that task. For classification: accuracy, macro-F1, a confusion matrix for both models and per-class errors. A class near zero for both usually means a wrong description. Routing a class to the model that is better at it makes a new classifier; score it on its own.
9. Give choice questions a way out. Offer a described
   option, or a separate yes/no question on whether any option fits, which TypeSafe's guidance prefers to a bare catch-all. Measure how often it fires and how often it is right.
10. Pick the cascade threshold on the development split from accuracy per confidence bucket and the cost of a wrong answer against an escalation. The threshold is policy; the accuracy you report comes from the test split. Check the final combined decision against labelled outcomes too, because answers about the same item tend to be wrong together.
11. Measure latency and cost as you will run them: same region, concurrency and caching. On Banking77 the frontier model cost about $19 per 1,000 requests uncached against $2.42 cached.

Example: a claims inbox, end to end

A home insurer receives first notice of loss by email. Each message must be routed to a queue, checked against the policy period and deductible, flagged if anyone was hurt, and acknowledged within the hour. The decision procedure, per judgment:

The routing call, in the documented shape (

The model answers every question on every call, so the order of checks is a gate in this function, readable and the same every time. The 0.5 and 0.8 cut-offs are policy: a missed injury costs more than a wasted injury-desk visit, so that threshold sits lower. Fit both on labelled history.

After an LLM drafts the acknowledgement, a second decision-model call checks it. The state holds the policy facts ("coverage is confirmed only after assessment") and the draft; one

Per claim that is two short decision-model calls, one LLM extraction if the email is unstructured, and one LLM draft. The decision-model calls are a small share of the latency and cost, and they are the ones that run on every message.

Comparison

Methods: the quality-scoring test

A compact record of how the 17-task numbers were produced.

• Tasks and items. 17 production LLM-as-a-judge evaluators in six categories, 170 held-out items in total (16 to 46 per category, as in the table above). Started from 23; six set aside as described above.
• Labels. Values from 0 to 1 taken from each evaluator's existing calibration set, which was not used to write or tune that evaluator's prompt or the decision-model questions.
• Questions. One
  question per task, written from the evaluator's prompt by an author who never saw the test items. The evaluator's few-shot examples were attached to the levels they illustrate. Four tasks had no examples, and neither side got any. Jev and Clef received identical request bodies. OpenAI Decisions received the same text translated field for field into its schema.
• Models. LLM judge: gpt-5.6-luna with the evaluator's full prompt. Jev pinned to
  . Clef 27B and Clef-flash 9B on Workers AI (no version string published). OpenAI Decisions with
  . Runs from 2026-10-04 to 2026-10-09.
• Scoring. Each decision-model
  was divided by its top level number to put it on the labels' 0 to 1 scale. Mean absolute error per task, then unweighted means over tasks for each category and overall. Pairwise ordering accuracy over pairs of items with different labels. Run-to-run SD is the spread between two passes of the same item, averaged per model.
• Latency. Wall-clock time per call from one client location. The judge's time includes about one second of client start-up. The decision models were called directly over HTTP.
• Not yet published. Who produced each calibration set and under which labelling instructions; how ties were counted in the ordering metric; the judge's sampling settings; whether the error figures average both passes; and item-level results. Until those are released, treat the scoring results as indicative.

Limitations

• Small samples. Banking77 is one domain, one prompt design and one fifteen-minute window, and short single-intent queries are a favourable shape. The scoring test has 170 items, 16 to 46 per category, so differences of 0.01 to 0.02 are within sampling noise. Our model-choice and wording findings come from 8 items and the calibration figure from 200. They show direction, not size.
• Possible memorisation. Banking77 (2020) and HaluEval (2023) may be in the models' training data, which would flatter scores undetectably.
• Labels from one source. Each scoring task's labels come from one evaluator's calibration set. Noisy labels make every model look worse; the label-average baseline is the check.
• Tasks selected on results. Four of 23 tasks were set aside because no model beat the baseline, so the 17-task figures describe the tasks where models did better than guessing the average.
• Ordering favours fine-grained output. The judge's 0.1 score steps cause ties that lower its ordering accuracy, so the ordering comparison mixes ranking ability with output format.
• Two passes per item. Enough to see low variation, not to show determinism or to estimate rare deviations.
• One LLM per test. Opus ran with reasoning off and no examples; either would probably raise its accuracy, latency and cost. A smaller judge would likely lose to Jev on several scoring tasks, and a larger one might widen the gap.
• Best-case cascade, criteria from label names. Banking77 thresholds were chosen on the test items, and both models lost a whole class to a description that could not be inferred from its name. Iterating descriptions on training data would lift both.
• Hosted Clef only. We did not test Clef's open weights on our own hardware.
• Versions move. These are Jev 1.13 results, not re-run on 1.14, and OpenAI Decisions is in beta. Results do not carry over between versions.
• Where a decision model is the wrong tool. Open answers, multi-hop reasoning, numbers, dates, counts, long noisy input, and anything someone must read the reasons for. TypeSafe says its model is not robust to instructions injected into the state, so it can be one layer of an injection screen, never the only one.
• Not a production design. Reading
  or
  straight from the API is fine for development. Production also needs versioned questions, calibration on your labels, and a rule for when to escalate to an LLM or a person.

Evidence and sources

Evidence:

• Third-party benchmark (Banking77 test split, n = 3,080, Jev 1.13 vs Claude Opus 5, run 2026-09-22): OpenRouter, Is Jev as Accurate as Frontier Models at Classification?
• Third-party benchmark (60 support tickets and 40 injection-screening messages, run 2026-09-19): OpenRouter, Jev vs LLM: When to Use a Decision Model Instead of Generating Text
• Product docs: Cloudflare, Clef and Clef-flash on Workers AI
• Product docs: OpenAI, Decisions guide
• Product docs: TypeSafe, Jev introduction, question primitives, Choice question type and HTTP API
• Our measurements at Scorable: quality scoring on 17 production evaluators, 170 held-out labelled items, 2 passes per item and model, runs 2026-10-04 to 2026-10-09, Jev pinned to
  , Clef and Clef-flash on Workers AI (no version string published), OpenAI Decisions with
  ; HaluEval-QA calibration, n = 200,
  , run 2026-09-19; readability ordering and wording sensitivity, 8 held-out items, run 2026-09-20. Not re-run on
  . Method summary in the Methods section above; item-level results not yet published.
• Updated: 2026-11-07

Further sources:

• Casanueva et al., 2020, Efficient Intent Detection with Dual Sentence Encoders (arXiv:2003.04807), the Banking77 paper.
• PolyAI, task-specific-datasets repository,
  (Banking77 train and test CSVs, CC BY 4.0).
• TypeSafe, "Jev: For the builders AND dreamers" (Jev 1.14 release post), 2026-10-14, and the Jev models page, for the 1.14 release and the
  alias.
• Li et al., 2023, "HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models", arXiv:2305.11747.
• Guo et al., 2017, "On Calibration of Modern Neural Networks", arXiv:1706.04599, for expected calibration error.
• McNemar, 1947, "Note on the sampling error of the difference between correlated proportions or percentages", Psychometrika 12(2).
• Efron and Tibshirani, 1993, "An Introduction to the Bootstrap", Chapman and Hall.
• Greshake et al., 2023, "Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection", arXiv:2302.12173.
• Kahneman, 2011, "Thinking, Fast and Slow", for the System 1 and System 2 framing behind the "System One" name.

FAQ

Is an LLM more accurate than a decision model? In these tests, slightly: 84.4 against 81.0 % on Banking77 classification, and a mean absolute error of 0.089 against 0.104 on 17 scoring tasks. Both are for one LLM in one configuration, and switching LLMs moved results by more than that.

Can a decision model replace an LLM classifier or judge? For closed labels and graded criteria, often, once you have checked it on your own labels. Keep the LLM for open answers, long mostly irrelevant input, or when you need the reasoning. The usual pattern is a cascade: the decision model on everything, the LLM on the uncertain share.

Is a decision model's confidence a probability? For a multi-class

Are decision models deterministic, and how much faster are they? We can't say they are deterministic. Clef and OpenAI Decisions returned identical scores across the two passes we ran, and Jev varied by an SD of 0.002, against 0.017 for the judge. That is low observed variation, and the wording of the question can still move results. They were typically about ten times faster: 0.18 to 0.74 s against 6.85 s in the scoring test, and 175 against 2,266 ms median on Banking77. These are indicative, not guarantees.

Can Cloudflare's Clef replace Jev? For some tasks. Clef-flash matched or beat Jev on tone and content safety, but over all 17 tasks Clef 27B and Clef-flash had error 0.145 and 0.141 against Jev's 0.104. Its advantage is open weights you can run yourself. Test few-shot examples per model: they helped Jev and OpenAI Decisions and hurt both Clef models.

How many labelled examples do I need? A few hundred give overall accuracy to within a few points; separating a 3-point gap needs a paired test and usually more than a thousand items. Aim for 30 to 40 per class for per-class error rates you can act on.

What should never go to a decision model? Arithmetic, date comparisons and counts (use code), anything that must produce new text or values (use an LLM), and security decisions where it would be the only line of defence. Checking an LLM's draft against a policy, on the other hand, is one of its best uses.

Related reading

• What is a decision model?
• Using LLM-as-a-judge for model evaluation
• How to measure and reduce noise in agentic LLM evals
• How to read and interpret classical LLM metrics vs. LLM-judge metrics
• How do you balance latency, cost, and precision in LLM systems?
• Tradeoffs between rule-based filtering and LLM moderation