Decision Models vs LLM-as-a-Judge: Jev, the OpenAI Decisions API, and When to Use Each

Updated: 2026-11-07 By: Ari Heljakka

Short answer

A decision model, also called a System 1 model (TypeSafe's term), answers a typed question (yes/no, one of several options, or a position on an ordered scale) with a probability and writes no text. Both names refer to the same thing; this post uses "decision model". An LLM judge writes a verdict and, if asked, the reasoning behind it. On closed criteria such as "is this answer supported by the passage", small public tests show the two agree with labels about equally often, while the decision model is typically around ten times faster, about five times cheaper, and gives probabilities you can threshold. On open rubrics that need reading comprehension across a whole document, the LLM judge still tracks expert ratings better. Most production setups should run a decision model on everything and send only the uncertain band to an LLM judge.

Key facts

• Definition: A decision model (or System 1 model) takes a state (the text to evaluate) plus typed questions and returns a probability per question, with no generated rationale. An LLM judge is a general model prompted with a rubric that returns a verdict as text or JSON.
• When to use: A decision model for closed criteria at full traffic volume, inline gating, and any place where a threshold on a probability drives an action. An LLM judge for open rubrics, multi-factor judgments, and whenever a person needs to read why.
• Limitations: No explanation, options must be defined in advance, results move with the wording of the question, and neither vendor publishes a calibration error. Independent calibration measurements vary widely by task.
• Example: A refund-reply check with three questions costs about $0.02 per 1,000 judgments and typically returns in under half a second; replies scoring between 0.2 and 0.8 go to an LLM judge for a written verdict.
• Jev 1.14: Released 2026-10-14; TypeSafe said
  would move to it a week later. The figures we measured in this post come from
  and have not been re-run on 1.14.
• Last updated: 2026-11-07

Key takeaways

• Agreement with labels is a tie on closed criteria. The differences are calibration, cost, latency and the missing rationale.
• The OpenAI Decisions API and Jev expose nearly the same three question types, but the first is a general model behind a constrained interface and the second is a model trained only for decisions.
• The model behind the LLM judge is the largest variable in any such comparison. A result that does not name it tells you little.
• A probability from either product is an estimate, not a guarantee. Check it against 50 to 100 of your own labelled items that you did not use to write the questions, and set thresholds from the cost of each kind of error.
• Use code checks first, a decision model second, an LLM judge third, and people for consequential novel cases.

Definition

An LLM judge is a general-purpose language model given a rubric and an item to grade. It generates tokens: a label, a score, usually a justification. Any confidence value it reports is itself generated text.

A decision model exposes a narrower contract. TypeSafe calls this class of model a "System One model", after the fast, intuitive System 1 of dual-process psychology, and you will see "System 1 model" and "decision model" used for the same thing. You send a state and a set of typed questions; it returns one typed answer per question with probabilities attached. There is no free text in the response, so there is nothing to parse and nothing to read.

Two hosted services currently offer this contract:

• Jev (TypeSafe, early access since 2026-09-15). TypeSafe describes it as a non-autoregressive model trained for calibrated decisions. Question types are
  (probability a statement is true),
  and
  . The current release,
  (2026-10-14), adds a
  type that returns a bounding box in an image, which is not a decision, and accepts PNG and JPEG images, an input TypeSafe says is not yet reliable enough for production.
• OpenAI Decisions API (public beta announced 2026-10-06,
  ). It runs
  , a general model, behind a decision-only interface. Question types are
  ,
  and
  . Text and inline image input.

Other decision models

Jev and the OpenAI Decisions API are the two this post compares, but they are no longer the only options. Several of the alternatives below accept Jev's request format, so trying one is mostly a change of endpoint and model name. The same request does not guarantee the same behaviour, so check each one on your own labels.

Open-weight and local:

• Clef and Clef-flash (Cloudflare, 2026-10-01). Open-weight decision models, 27B and 9B parameters, under the Apache 2.0 license. They run on Workers AI or your own hardware and follow the System One request format. In our comparison on 17 quality-scoring tasks, Clef 27B and Clef-flash had a mean absolute error of 0.145 and 0.141, against 0.104 for Jev 1.13 and 0.089 for an LLM judge. They beat Jev on tone and content safety and fell furthest behind on readability and on comparing text with a source.
• Laya (Convai Innovations). A small non-autoregressive model that scores every option in one forward pass. There is an English checkpoint on ModernBERT-large (421M parameters) and a multilingual one on mmBERT (322M). Its model card reports about 33 to 40 ms per question on a T4 GPU. Its zero-shot accuracy is weak, so plan to fine-tune it.
• Kev (open source, Apache 2.0). A family of decision models inspired by Jev, built on Qwen bases at 0.8B, 4B and 9B, with a 27B model added in Kev 1.0. Sized for local and edge hardware.
• Foq. A local, open-source decision engine built on a 1.58-bit 8B model (2.2 GB). It reports about 25 ms per decision on a consumer GPU. It also runs on CPU, but publishes no CPU timing.
• NanoJev. A 0.6B Qwen-based model trained to choose actions in simple games (maze, snake, ViZDoom). It shows how small a decision model for a control loop can be.
• SemIf. Trains no new model. It reads the probability of each option directly from an existing open LLM in a single forward pass, then applies per-workload temperature scaling to correct calibration.
• OpenJev. An open-source adapter that puts general chat LLMs on fast inference providers such as Groq behind Jev's request format. Expect LLM-like cost and accuracy with a decision-model interface.

Hosted:

• OpenAI Decisions API, described above.
• Microsoft-Decision-1 (Microsoft Foundry, public preview since 2026-10-09). A hosted decision model for classification, routing and evaluation. It is not open-weight and has no published calibration figures, and Microsoft's own advice is to validate calibration on your data. Microsoft also exposes a System One-style endpoint for it.
• Levanto Sage 2 (Levanto Labs). A hosted decision model whose
  endpoint accepts Jev's request format, without some of Jev's features. It is aimed at automated workflow routing.

When this matters

• You want to evaluate every production trace, not a 1 % sample, and the per-item cost of an LLM judge is what limits coverage.
• The evaluator sits inline (gating a tool call, a reply, or a cheaper model's draft) and a 1.5 second judge call is too slow.
• A downstream rule needs a number it can threshold, and the judge's self-reported confidence only ever takes a handful of values.
• You need many criteria on the same item. Both services answer all questions in one request against the same input.

It matters less when the rubric is open-ended, when the volume is low, or when the output of the evaluation is a written review for a human.

How it works

Two different kinds of answer

An LLM judge returns something like

Agreement is a tie on closed criteria

In a published test on 88 relabelled HaluEval question-answer items, Jev matched the labels on 84 and an LLM judge (a small, fast general model at temperature 0) on 83. Our own run at Scorable on 200 HaluEval-QA items (2026-09-19,

Calibration is where they differ

In the same published test the Brier score was 0.043 for Jev and 0.054 for the LLM judge, whose stated confidence took only six distinct values. Independent tests now cover far more than that one dataset. Together they show that how well Jev's probabilities hold up depends on the task. Expected calibration error (ECE) is the average gap between stated probability and observed frequency, so lower is better.

• Good on narrow, factual yes/no questions. ECE was 0.012 to 0.046 across nine variables on 150,000 police crash narratives (Rafe and Das, 2026), 0.011 on 400 claim-verification items (Vals.ai), and 0.037 in our own HaluEval-QA run (10 bins, n = 200).
• Worse on subjective, legal or rating tasks. ECE was 0.108 on LegalBench, the worst of the 12 models Vals.ai tested, a median of 0.157 over 15 social-science annotation tasks (Ibrahim and Zaki, 2026), and 0.284 on HelpSteer2 answer rating (AI/ML API).
• Better than most LLMs' stated confidence, but not the best. In the social-science study, Jev's probabilities were better calibrated than the self-reported confidence of 16 of 19 LLMs, but worse than the three strongest frontier models. Items it scored above 0.9 were correct only about 82 % of the time.
• Recalibration helps. In the crash-narrative study, a simple recalibration on labelled data cut calibration error by a factor of 3.3.

TypeSafe states the model is trained to return calibrated decisions but has published no calibration figure. Its own documentation says score levels are "weak in numerical calibration" and that choice questions lean towards the option listed first. OpenAI's documentation makes no calibration claim and tells you to set thresholds from your own labelled examples.

Developers testing the Decisions API beta reported that

Cost and latency

Treat every latency figure in this post as indicative: a typical median from a small test over the public internet, not a guarantee. Your numbers will move with region, payload size, the number of questions per request, concurrency and the provider's load on the day. Measure them from where your code runs.

The published test measured $0.021 against $0.114 per 1,000 judgments and a median of 171 ms against 1,662 ms. Our measurement was about $0.022 per 1,000 and a 0.42 s median. Both decision services bill input tokens only; at list price the Decisions API costs about 2.4 times as much per input token as Jev, which matches an early developer report of 2 to 3 times on identical questions. OpenAI states the Decisions API is about ten times faster than the same model through its Responses API and gives no absolute figure.

Bias

LLM judges are known to favour the first and the longer answer (Zheng et al., 2023). In the published test neither judge changed its pick across 72 reordered pairs. Padding a wrong answer with two true sentences fooled the LLM judge on 4 of 36 items and Jev on 2 of 36. The sample is too small to separate them.

Where the LLM judge wins

On 50 SummEval summaries rated by experts, the LLM judge's consistency scores correlated with the experts at Spearman 0.72 and Jev's at 0.47. Coherence was a tie at 0.44 and 0.42. Judging whether a summary is consistent with a long article means holding many facts at once, which is the case TypeSafe's own guidance says to split into narrow questions.

The gap also depends on which LLM sits behind the judge. On a readability-ordering task we ran, the same judge rubric scored 0.786 pairwise ordering accuracy on a gpt-4o-class model and 0.964 on gpt-5.5, while Jev scored 0.929 to 1.000 depending on how the levels were worded (8 held-out items, so indicative only).

Example: a refund reply checked three ways

A support agent drafts a reply. Before it is sent you want to know three things: is it grounded in the refund policy, does it promise something the policy does not allow, and should a person look at it.

The Jev request, following the documented shape:

The same check against the Decisions API uses

Routing on the result:

In our HaluEval-QA run, acting only outside the 0.2 to 0.8 band decided 80 % of items at 95.0 % accuracy and left 20 % for the slower judge. Note the phrasing: each

The other two questions are combined in the same function, not in the model. The decision model answers every question in the request regardless of the others, so a rule such as "an ungrounded reply is never sent, whatever else it scores" belongs in your code after the call, where it is a few visible lines. For

Comparison

Which judge for which rubric

1. A regex, schema or exact-match check can decide it: use code.
2. A closed criterion with predefined answers, at volume or inline: use a decision model.
3. An open rubric, a multi-factor judgment, or a required explanation: use an LLM judge.
4. A consequential case unlike anything in your labelled set: use a person.

Limitations

• No rationale. When a decision-model score is wrong you cannot read why. Debugging means changing the question wording and re-running against labels.
• Wording sensitivity. In our tests the same question design moved held-out error from 0.038 to 0.079 by rewording the level descriptions, and few-shot examples helped one wording and hurt another. A reworded question is a new question: give it a new version and measure it again.
• Narrow questions only. Counting, numeric and date comparison, multi-hop reasoning and double negation are documented weak spots for Jev. Compound conditions should be split and combined in code.
• Task-dependent calibration. No vendor-published calibration error exists for either service. Independent measurements of Jev range from an ECE of 0.011 on factual claim checks to 0.284 on answer rating.
• Platform risk. Both are hosted, early-stage endpoints with moving model aliases. TypeSafe announced on 2026-10-14 that
  would move from
  to
  a week later, so a client sending the alias gets a different model from the one our figures were measured on without any change to its code. Pin a version, store the question version and model version with every score, and re-check your thresholds when either changes. An agreement figure holds only for the question version, model version and label set it was measured on.
• Not a replacement for labels. A calibrated-looking probability still needs to be checked against your own human judgments before it gates anything.

Evidence and sources

Evidence:

• Third-party benchmark (88 HaluEval items, 50 SummEval summaries, run 2026-09-21): OpenRouter, Jev vs LLM-as-a-Judge
• Product docs: OpenAI, Decisions guide
• Product docs: TypeSafe, Jev introduction
• Our measurements: HaluEval-QA, n = 200,
  , run 2026-09-19; readability ordering, 8 held-out items, run 2026-09-20. Not re-run on
  . Methodology write-up pending.
• Updated: 2026-11-07

Further sources:

• TypeSafe, "Jev: For the builders AND dreamers" (Jev 1.14 release post), 2026-10-14, and the Jev models page, for the 1.14 release, image input, long context and the
  alias.
• Li et al., 2023, "HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models", arXiv:2305.11747.
• Fabbri et al., 2021, "SummEval: Re-evaluating Summarization Evaluation", arXiv:2007.12626.
• Zheng et al., 2023, "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena", arXiv:2306.05685.
• OpenAI Developer Community, "Decisions API is now available in Public Beta", 2026-10-06, for the beta announcement and early developer reports.
• TypeSafe, System One concept and model jaggedness documentation, for the term and the documented weaknesses.
• Rafe and Das, 2026, calibration of Jev on police crash narratives, arXiv:2609.24052.
• Ibrahim and Zaki, 2026, Jev on 15 social-science annotation tasks, arXiv:2609.24574.
• Vals.ai, Independent evaluation of Jev, 2026-10-06.
• AI/ML API, What is Jev, for per-task ECE on TweetEval, Banking77 and HelpSteer2.
• Alternatives: Cloudflare Clef, Laya, Kev, Foq, NanoJev, SemIf, OpenJev, Microsoft-Decision-1, Levanto Sage 2.

FAQ

Is Jev calibrated? On narrow, factual yes/no questions, mostly yes. Independent tests find an expected calibration error of about 0.01 to 0.05 on such questions, including 150,000 crash narratives in one study and 0.037 on 200 items in our own run. On subjective, legal or rating tasks it is several times worse: 0.108 on LegalBench, a median of 0.157 across 15 social-science annotation tasks, and 0.284 on answer-quality rating. TypeSafe has published no figure and its own documentation warns that score levels are weak in numerical calibration. Use the probability directly only on simple factual checks you have tested. Elsewhere, treat it as a ranking until you have checked it on your own labels. A simple recalibration on those labels cut calibration error by a factor of 3.3 in one study.

Is the OpenAI Decisions API a decision model or an LLM judge? It is a general LLM restricted to answering typed questions. You get the decision-model contract (probabilities, no text, input-only billing) with the behaviour of the underlying general model, including image input.

What are the alternatives to LLM-as-a-judge? Deterministic code checks, decision models, fine-tuned classifiers, and human review. They are layers, not substitutes.

When should I keep an LLM judge? When the rubric is open, when several factors must be weighed together, when a person will read the explanation, or when the decision model's probability falls in your uncertain band.

How do I calibrate a decision model on my own data? In four steps:

1. Label 50 to 100 items that look like your real traffic.
2. Run your questions on them and record each probability.
3. Check the probabilities. Group the items by predicted probability (0 to 0.1, 0.1 to 0.2, and so on) and count how often each group is actually "yes". If the model is well calibrated, items it scored around 0.7 are "yes" about 70 % of the time.
4. Pick two cut-offs. Above the upper one, block automatically. Below the lower one, let the item through. In between, send it to an LLM judge or a person. Where you put them depends on which mistake costs you more: letting a bad item through, or blocking a good one.

Use items you did not look at while writing the question or its examples, and repeat the check whenever the model version or the question wording changes. The full procedure, with code, is in how to calibrate a decision model.

Can I run a decision model and an LLM judge side by side? Yes. Evaluation platforms now store each typed answer as a score on the trace next to LLM-judge and code-evaluator scores, so a common pattern is the decision model on every observation and the LLM judge on a sample or on flagged items.

Related reading

• Using LLM-as-a-judge for model evaluation
• LLM as a Judge vs. Human Evaluation
• What is a decision model?
• Decision model vs LLM: accuracy on scoring and classification, and when to use each
• How to calibrate a decision model and use it to catch conversation signals