System One decision models: Jev and the case for classifiers

Many decisions inside an AI system are choices from a short list, such as which tool to call or whether to allow an action. Early third-party tests of TypeSafe's new Jev model, along with years of work on classifiers, show that purpose-built decision models can make many of these choices at a small fraction of a frontier model's cost and wait time. They are not more accurate by default, so the real work is measuring them on your own labeled data and deciding what happens when they are unsure.

Many decisions in an AI system are choices

Follow one request through a typical agent. Something picks the workflow, then a tool. Something checks whether a shell command or a refund is safe to run. Something screens the answer for leaked data, and an evaluation job later scores the run.

Each step picks from a fixed set of answers: yes or no, one of 77 intents, allow or block, a score from 1 to 5. None of them needs an essay.

Many teams still send each of these steps to a frontier large language model (LLM), ask for a label, and parse the text that comes back. When the step is a grade or a verdict, this is usually called LLM-as-a-judge. It works, but it is slow and costly. In one recent test, Claude Opus 5 at high reasoning effort took over 3 seconds and cost $14.30 per thousand judgments to flag answers that were not supported by their sources [1].

Even a frontier lab turns its own LLM into a classifier for this job. Anthropic's Claude Code auto mode, which approves or blocks an agent's actions in place of a human, starts with "a fast single-token filter," a one-word yes or no from Sonnet 4.6, and runs step-by-step reasoning only when that filter flags something [2].

A decision model is the purpose-built version of that idea. It takes an input and returns one answer from a set you define, with a probability attached. Classifiers, routers, guardrails and judges are all decision models under different names. TypeSafe calls its version a System One model, after the term Daniel Kahneman popularized for fast, intuitive thinking [3].

Jev: what TypeSafe claims, and what third-party tests found

TypeSafe announced Jev on September 15, 2026 [3]. An application sends it context, such as text or JSON, and a set of questions [4][5]. Jev "evaluates those questions in parallel and returns typed choices, scores, or true-or-false answers, along with probabilities" [5]. It does not write prose [3].

TypeSafe says end-to-end response time is 70 to 500 milliseconds, that Jev can be "40x-200x faster for the same levels of frontier intelligence," and that it "can't hallucinate" [3]. Input costs $0.042 per million tokens and output is free [3]. The company is candid that its headline speed and cost figures are "on the higher end of real world gains," and that its reference answers average two frontier models, "which biases answers towards OpenAI and Anthropic's models" [3]. "Can't hallucinate" means Jev cannot return an answer outside the set you defined. It can still return the wrong one.

Vercel reports that within 24 hours, nearly 13% of paid teams on its AI Gateway were using Jev, twice the share of the GPT-5.6 family [5]. That measures trial, not production results.

Third-party tests, all published within three weeks of launch and mostly on public data, tell you more. None of these testers is TypeSafe, but most have a stake: Arize offers Jev in its evaluation tools [6], OpenRouter sells access to it [7], and Red Hat will ship rival guardrails [9]. Raschka says he is unaffiliated [8]. These tests all predate Jev 1.14, released October 14, 2026, which TypeSafe describes as small improvements with the largest gain expected in languages other than English [4][21].

TypeSafe's absolute speed and price claims hold: median response times ran from about 130 to 360 milliseconds, and costs were tens to hundreds of times below a frontier judge. The high end of those times came from Red Hat's test, run from the UK against TypeSafe's US service [9][22]. TypeSafe's speed multiple does not hold in these tests, which measured about 13x to 30x against frontier models rather than 40x to 200x.

Accuracy depends on the task and the threshold. Jev matched Opus 5 on hallucination detection only after Arize tuned its cut-off [1], and it trailed by 3.3 points on banking intents [7]. Red Hat concluded that decision models like Jev "do not reliably outperform LLM-as-a-judge, pre-trained predictive models, or open source decision models in speed or accuracy" [9]. Red Hat tested self-hosted open LLMs, not frontier judges, and one of them, Qwen3.6-35B, beat Jev on injection accuracy and latency. Jev ranked first of nine on safety and fourth of nine on injection [9]. Red Hat will ship the DeBERTa and Granite Guardian classifiers as its default guardrails in OpenShift AI 3.6, so weigh that the way you would weigh a vendor claim [9]. The same tests found open alternatives can compete with Jev, unevenly: DiffusionGemma stayed within 1.4 points on both tests, while Laya was within a point on injection but scored 57.87% on safety against Jev's 86.20%, and 75.20% after prompt tuning [9].

The probabilities rank answers well, but they are not calibrated out of the box: OpenRouter found Jev's score overestimated its accuracy in the mid-range [7], and Arize found it underconfident [1]. In Arize's ten-run test, "All 3,561 runs where Jev put at least 99% on its answer were correct" [6]. Those runs repeat at most 517 examples, and each judge missed only 4 to 14 of them, so Arize treats this as a direction rather than a precise ranking [6]. On banking intents, Jev was 96.3% accurate on the 58% of messages where its confidence was at least 0.99, and 29.6% accurate on the 3.5% below 0.5 [7].

Open questions remain. TypeSafe has not published Jev's architecture or training data; Raschka guesses it is "something small similar to ModernBERT" given its speed [8]. TypeSafe's launch FAQ disputes the size part, saying Jev is neither small nor an LLM, and says the company makes all its training data itself [3]. One model serves every customer: TypeSafe says Jev is not fine-tuned, or further trained, on customer data, and that English is where "accuracy is currently best" [4]. TypeSafe says Jev is currently available only through its hosted API, with all data processed in the US [22]. TypeSafe's own docs list weak spots for Jev 1.13, and it has not published an equivalent list for 1.14. Jev "is not a calculator," "does not count reliably," reads dates as text, sometimes leans toward the first option in a list, and can be moved by "content written to adversarially steer the model" [10]. Octomind showed this in one test: a planted note saying a command that deletes SSH keys was pre-approved cut Jev's block probability from 0.76 to 0.48 and its confidence from 0.64 to 0.22, though the verdict stayed "block" [11]. A September 2026 preprint found the same pattern across 510 cases: planted content shifted Jev's probabilities but rarely made it choose the attacker's target, with success on fresh calls rising from 1.8% to 3.5% under adaptive attacks [23]. HiddenLayer also got an injection past a Jev-driven browser agent, though less often than against a Qwen model [24]. And Arize notes that its datasets "are public and predate every model tested," so the models may have seen them in training [1]. Raschka and OpenRouter flag the same training-data risk for IMDb and Banking77 [7][8].

Classifiers never went away

Sorting text into categories is one of the oldest jobs in machine learning. Raschka traces it from bag-of-words spam filters, which count words and ignore their order, up to Jev, and still calls bag-of-words plus logistic regression, a simple statistical classifier, "my go-to baseline for every text classification problem" [8].

The modern workhorse is the encoder. An encoder reads the whole input at once and turns it into a numeric summary that a small final layer can label. It does not write text word by word, which is why it is fast. The ModernBERT authors call encoders "the workhorse of numerous production pipelines" and released an updated one in December 2024, trained on 2 trillion tokens with an 8,192-token input window [12]. The Ettin study, accepted at the ICLR 2026 conference, trained encoders and text-generating decoders with the same recipe and found encoders better at classification and retrieval: "a 400M encoder outperforms a 1B decoder on MNLI," a standard test of whether one sentence follows from another [13].

Safety teams have used purpose-built classifiers for years. Meta's Llama Guard 4, a 12-billion-parameter model that runs on a single GPU, labels prompts and responses safe or unsafe and lists the categories violated [14]. ShieldGemma, built on Gemma 2, screens prompts and outputs for harms such as hate speech and dangerous content [15]. IBM's Granite Guardian covers jailbreaks, meaning prompts that trick a model past its safety rules, and answers that are not supported by retrieved text [16]. The "JurEE not Judges" paper made the case directly: an ensemble of small encoders that returns probabilistic risk scores, with thresholds users set themselves, instead of an LLM judge's text verdict [17].

The clearest recent signal comes from Anthropic. Its Constitutional Classifiers++ system, described in a January 2026 preprint, uses "a two-stage classifier cascade where lightweight classifiers screen all traffic and escalate only suspicious exchanges to more expensive classifiers." Anthropic reports that the cascade, linear probes and an ensemble together cut compute cost 40x against its baseline exchange classifier, with a 0.05% refusal rate on production traffic [18].

Agents explain the timing. Every tool call and handoff is another small decision, and each one is a cost to cut or a check worth adding.

The trouble with frontier models as judges

LLM-as-a-judge is flexible and quick to set up. At scale it has real problems.

Cost and speed come first. The tests above put a top model at seconds per call and about $2 to $17 per thousand calls, depending on settings: $2.42 is Opus 5 with reasoning off and a cached prompt, and $14.30 is high reasoning effort [1][6][7]. Multiply that by every step of every agent run.

Answers can also shift between runs, though that depends on the model more than its size. In Arize's ten-run test, the small GPT-5 nano changed its answer on 7.9% of examples and the frontier GPT-5.6 Sol on 3.3%, while Opus 5 changed on 0.77%, level with drop-in Jev at 0.97%; Jev through its native API was lowest at 0.19% [6].

Judges have biases. A 2026 paper in Transactions on Machine Learning Research, testing five older judge models from four providers, found that "style bias is the dominant bias," with judges favoring markdown over plain prose, far ahead of the better-known position bias [19].

Stated confidence needs checking. Asked for a confidence score on Arize's hallucination test, GPT-5.6 Terra averaged 96% and was right 70% of the time. Opus 5 at high effort said 81% and was right 83%, better calibrated than Jev, which was underconfident [1]. Calibration varies by model, so you have to measure it.

There is also an audit gap. A paragraph of reasoning reads like an explanation, but an incident review needs a logged input, a fixed label set, a probability, a threshold and a measured error rate.

Calibrated confidence is what makes a decision usable

A model is calibrated when its probabilities mean what they say: of all the answers it gives at 90%, about 90% are right. Then software can act on the number. You pick a threshold, the cut-off above which the system acts on its own, and send everything below it to a bigger model or a person.

Thresholds must be tuned on your data. At the default cut-off of 0.5, Jev scored 76% on Arize's hallucination test, partly because "34% of perfectly grounded responses score above 0.5." At a cut-off of 0.80 it matched Opus 5 at 87% [1]. In Jev, probability and confidence are different numbers, so a cut-off tuned on one does not carry over to the other [28].

Escalation can recover most of a gap. In OpenRouter's banking test, "Routing everything under 0.9 confidence to Claude Opus 5 recovered 84.0% accuracy, within 0.4 points of Opus alone," at $0.69 per thousand requests instead of $2.42, with 76% of traffic never reaching Opus [7]. OpenRouter calls this an upper bound, because it was measured on the same examples used to report accuracy [7]. Escalation is not always worth it, though. In Arize's test, once the threshold was tuned, escalating any share of Jev's judgments to Opus 5 moved accuracy by less than a point [1].

Anthropic's auto mode shows how to tune a gate to the cost of each mistake. Its first stage is told to "err on the side of blocking," and wrongly flags 8.5% of a 10,000-item sample of Anthropic's internal traffic. The second stage brings that to 0.4% [2]. The full pipeline still misses 17% of real overeager actions in a set of 52, which Anthropic calls "the honest number" [2]. After 3 consecutive denials or 20 in a session, it stops and hands control to a person [2]. Publish the miss rate, and design the fallback around it.

Where decision models fit, and when to train your own

Decision models belong wherever software needs a choice rather than prose.

Next, choose what kind of model makes the call.

Train your own when a decision is high-volume and stable, and when the data is private or must stay on your servers. A June 2026 preprint illustrates the gap. Its best sub-billion model, a 0.5-billion-parameter model fine-tuned for relation extraction (pulling structured facts out of text), scored 0.83 F1, a combined precision-and-recall score, on general-domain benchmarks. GPT-5.4 scored 0.69 and Claude Sonnet 4.6 scored 0.66 without task training [20]. An in-domain RoBERTa encoder also beat both. The authors stress that small models are not stronger in general; the gain "stems from task adaptation rather than generative decoding" [20]. Red Hat reached a related view on off-the-shelf guardrails: "pre-trained predictive models remain extremely competitive" [9]. Our companion article on small language models covers this in depth.

Labeled data is the usual blocker, and large models can help produce it. ShieldGemma and Granite Guardian both used synthetic data [15][16], and JurEE used LLM-assisted augmentation [17]. A practical pattern: have a frontier model or a hosted decision model draft labels, have people check a sample, then train a small classifier you own.

Hosted decision models make the most sense when you have little labeled data or many different questions to ask. Red Hat's conclusion is that for guardrail cases "where pre-trained models are unavailable or insufficient," zero-shot classifiers, meaning models used without task-specific training, are "a capable alternative to LLM-as-a-judge" [9]. Jev is no longer the only hosted option: OpenAI's Decisions API became generally available in October 2026 [25], and at its October 13, 2026 launch Artificial Analysis's Decision Index ranked Jev 1.13 first among decision models at 58, behind the LLM Claude Haiku 5.5 (Low) at 61, which was about 11 times slower [26]. By November 1, 2026, five decision models ranked ahead of Jev on the live index, led by Liquid AI's d1-ultra at 65 [27].

An owned classifier needs monitoring. A weekly shift in the label mix or the spread of confidence scores is often the first sign that inputs have changed. Re-label a small random sample each month and retrain when accuracy falls below target.

What to do next

1. List your decisions. Find every point where your system picks from a fixed set. For each, note volume, latency budget, and the cost of a false positive (acting when it should not) against a false negative (missing what it should catch).

2. Build a labeled set. Arize suggests: "Get a few hundred examples of your own traffic labeled by people. Run your judge over them, but keep the probability rather than the label" [1]. Hold part of it back for testing.

3. Compare candidates on the same examples. Include your current LLM judge and at least one cheaper option, such as a hosted decision model or a small encoder fine-tuned on your labels.

4. Set a threshold and an escalation path for each decision. Decide who or what handles the uncertain cases, and how fast.

5. Treat inputs as hostile. Test every gate with injected instructions [10][11]. Never let one model be the only check on a destructive action; pair it with fixed rules, such as an allow-list of approved commands.

6. Log every decision. Record the input reference, model version, label, probability, threshold and outcome. That log is your audit trail and your next training set.

What to measure, on your own labels:

• Accuracy, false positive rate and false negative rate at your chosen threshold.

• Calibration: whether answers given at 90% confidence are right about 90% of the time.

• Escalation rate, and how often reviewers overturn the model.

• Latency: the median, and the slowest 5% of calls (95th percentile).

• Cost per 1,000 decisions, including escalations.

• Stability: how often the same input gets a different answer on a rerun.

• Drift: week-to-week change in label mix and confidence spread.

How we think about this at OctaNeural

We treat most decisions inside an agent as classification problems, each with a price and an error rate. That fits the pattern in this article: a small model first, and a bigger model or a person when confidence is low. Where volume justifies it, we train compact classifiers on a client's own data and agent traces as part of a custom deployment, with ownership set by each agreement, so weigh our view with that in mind. If you are working on this problem, bring it to us.

Sources

1. Laurie Voss (Arize AI). "Jev vs. LLM-as-a-Judge: Accuracy and cost benchmarks." Arize blog, September 2026. https://arize.com/blog/jev-as-a-judge/

2. John Hughes (Anthropic). "How we built Claude Code auto mode: a safer way to skip permissions." Anthropic Engineering, March 2026. https://www.anthropic.com/research/claude-code-auto-mode

3. Diogo Almeida (TypeSafe). "Introducing System One Models & Jev." TypeSafe blog, September 2026. https://typesafe.ai/blog/introducing-system-one-models-and-jev

4. TypeSafe. "Models." TypeSafe documentation, accessed November 2026. https://docs.typesafe.ai/models

5. Amelia Charles, Harpreet Arora, Eric Dodds (Vercel). "Jev is the fastest-adopted model in AI Gateway history." Vercel blog, September 2026. https://vercel.com/blog/ai-gateway-jev-model-launch

6. Elizabeth Hutton (Arize AI). "What Jev's Probabilities Reveal That Repeated LLM Judgments Miss." Arize blog, September 2026. https://arize.com/blog/jev-llm-judge-consistency/

7. Kenny Rogers (OpenRouter). "Is Jev as Accurate as Frontier Models at Classification?" OpenRouter blog, September 2026. https://openrouter.ai/blog/insights/jev-vs-claude-opus-5-classification/

8. Sebastian Raschka. "Language Models for Text Classification: From Bag-of-Words to Jev." Ahead of AI, September 2026. https://magazine.sebastianraschka.com/p/classifier-history-and-jev

9. Rob Geada, Mac Misiura, Shelton Cyril (Red Hat). "Benchmarking AI decision models against traditional guardrails." Red Hat Developer, October 2026. https://developers.redhat.com/articles/2026/10/02/benchmarking-ai-decision-models-against-traditional-guardrails

10. TypeSafe. "Jev 1.13 jaggedness." TypeSafe documentation, accessed November 2026. https://docs.typesafe.ai/model-jaggedness/jev-1.13

11. Don Karter (Octomind). "Jev Explained: TypeSafe's System One Model and the Decisions Inside Every AI Agent (2026)." Octomind blog, September 2026, updated October 2, 2026. https://octomind.run/blog/jev-system-one-model-ai-agents

12. Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, et al. "Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference." arXiv:2412.13663, December 2024. https://arxiv.org/abs/2412.13663

13. Orion Weller, Kathryn Ricci, Marc Marone, Antoine Chaffin, Dawn Lawrie, Benjamin Van Durme. "Seq vs Seq: An Open Suite of Paired Encoders and Decoders." ICLR 2026 (arXiv:2507.11412), July 2025, revised March 2026. https://arxiv.org/abs/2507.11412

14. Meta. "Llama Guard 4 12B" model card. Hugging Face, April 2025. https://huggingface.co/meta-llama/Llama-Guard-4-12B

15. Wenjun Zeng, Yuchi Liu, Ryan Mullins, et al. "ShieldGemma: Generative AI Content Moderation Based on Gemma." arXiv:2407.21772, July 2024. https://arxiv.org/abs/2407.21772

16. Inkit Padhi, Manish Nagireddy, Giandomenico Cornacchia, et al. "Granite Guardian." arXiv:2412.07724, December 2024. https://arxiv.org/abs/2412.07724

17. Dom Nasrabadi. "JurEE not Judges: safeguarding llm interactions with small, specialised Encoder Ensembles." arXiv:2410.08442, October 2024. https://arxiv.org/abs/2410.08442

18. Hoagy Cunningham, Jerry Wei, Zihan Wang, Andrew Persic, et al. "Constitutional Classifiers++: Efficient Production-Grade Defenses against Universal Jailbreaks." arXiv:2601.04603, January 2026. https://arxiv.org/abs/2601.04603

19. Sadman Kabir Soumik. "Judging the Judges: A Systematic Evaluation of Bias Mitigation Strategies in LLM-as-a-Judge Pipelines." Transactions on Machine Learning Research, 2026 (arXiv:2604.23178), April 2026. https://arxiv.org/abs/2604.23178

20. Despina Christou, Grigorios Tsoumakas. "Sub-Billion, Super-Frontier: Small Language Models Rival Zero-Shot Frontier LLMs on General and Literary Relation Extraction." arXiv:2606.22606, June 2026. https://arxiv.org/abs/2606.22606

21. TypeSafe. "Jev: For the builders AND dreamers." TypeSafe blog, October 2026. https://typesafe.ai/blog/jev-1-14

22. TypeSafe. "Enterprise." TypeSafe website, accessed November 2026. https://typesafe.ai/enterprise

23. Tiantong Wu, Wei Yang Bryan Lim. "Decision Hijacking: Prompt Injection Attacks on Jev's Typed Probabilistic Decisions." arXiv:2609.28613, September 2026. https://arxiv.org/abs/2609.28613

24. Thomas Claburn. "Jev and other decision models can still be tricked with prompt injection." The Register, October 2026. https://www.theregister.com/ai-and-ml/2026/10/15/jev-and-other-decision-models-can-still-be-tricked-with-prompt-injection/5303184

25. OpenAI. "Changelog." OpenAI API documentation, accessed November 2026. https://developers.openai.com/api/docs/changelog

26. Artificial Analysis. "Artificial Analysis Decision Index v1.0." Artificial Analysis, October 2026. https://artificialanalysis.ai/articles/artificial-analysis-decision-index

27. Artificial Analysis. "Artificial Analysis Decision Index." Live leaderboard, accessed November 1, 2026. https://artificialanalysis.ai/evaluations/artificial-analysis-decision-index

28. TypeSafe. "Confidence." TypeSafe documentation, accessed November 2026. https://docs.typesafe.ai/confidence

Sources checked November 1, 2026. Test results are for Jev 1.13.