Early independent tests of a new decision model, and what they show about calibration and thresholds in production.

A GitHub issue arrives: after upgrading to v3.2, login occasionally returns 502 when the session expires. Somebody has to decide which team owns it. Authentication, API Gateway, Client, or Infrastructure.

A language model can make that decision today. With an enum schema, structured-output modes can hold it to a fixed set of values, but the answer is still generated token by token. If you want a probability for the decision, one common way is to derive it from token log-probabilities, where the API exposes them.

On September 15, 2026, TypeSafe AI released Jev in early access and proposed a different shape. They call it a System One Model. You send a state and a set of typed questions, and you get back answers drawn from options you defined in advance, each with a probability attached.

No generated text to parse, and no answer outside the schema.

Those are per-option probabilities. For choice and score questions, Jev also returns a separate confidence field, a single number summarizing how concentrated the probabilities are. They are different quantities. Below, probability means the number attached to an answer, the confidence field means Jev’s separate value, and score means whichever of the two a policy reads.

Your routing rule will read that 0.71 and act on it. This post is about what has to be true of that number for the rule to work, who is in a position to know, and what early independent tests of Jev found.

Speed is not the interesting part

According to TypeSafe, Jev produces all of its outputs in a single parallel query, answers in 70 to 500 milliseconds, and costs $0.042 per million input tokens with output free. Their headline multipliers, 193.6 times faster and 444.6 times cheaper, come from their own workflow evaluations, which they expect to be on the higher end of real-world gains. I have not run Jev, so I will not argue about speed. The more interesting question is what your code does with the probability, and that applies to any model that hands you a number to act on.

Constrained output guarantees validity, not correctness

TypeSafe says Jev cannot hallucinate. In their evidence section they add that Jev’s 0 percent structured-output and tool-call error rates are not empirical: because schema matching is guaranteed, they can place 0 percent in the plots with confidence.

That is a good disclosure, and the nuance matters more than the zero. Jev cannot produce an answer outside the declared options. It will not return a fifth option, an empty string, or a paragraph of apology where a label was expected. That guarantee is about output-space validity, not truth, and a schema-valid answer can still be false. An independent study on synthetic support tickets, discussed below, was consistent with it: zero invalid responses in 4,621 calls.

Being wrong is still possible in two ways. The first is misclassification inside the set. If the right owner was API Gateway, Authentication at 0.71 is valid, the most probable option, and wrong, and nothing in the type system notices. Schema enforcement removes one loud failure mode. The errors that remain are quieter: a wrong route can be perfectly well formed, pass every interface check, and surface three days later in someone’s sprint retro. The same study’s raw results contain the extreme case 17 times: 1.00 on a wrong answer and 0.00 on the right one.

The second is the option set itself. If the real answer is a team you did not list, the model still has to pick one of the four. In one early probe, a forced choice between two options, neither of them valid, returned its top answer at 0.52 with a confidence field of 0.04, so the distribution flagged it, but that is one probe, and its raw response was not kept. TypeSafe’s guidance is to include a no-match outcome whenever nothing may fit. Watch how often it gets chosen, because a rising rate is one of the cheapest signals that the list has gone stale.

A closed output space removes format failure. A missing option needs a design change. For misclassification inside the set, the model exposes a probability distribution over the offered answers, but whether that distribution says anything reliable about semantic error is something the surrounding system still has to establish.

What a calibration claim actually says

TypeSafe’s comparison table glosses calibration as “higher confidence means higher accuracy.” The wording invites confusion, because the API also returns a field literally named confidence, which TypeSafe’s guidance describes as distribution concentration, “not overall workflow correctness or permission to act.” The documentation’s definition of calibration is more precise: outcomes given a probability of 0.8 should happen about 80 percent of the time, and outcomes given 1.0 every time. Those are two different claims.

The first is about ordering. Sort predictions by probability and accuracy rises as you go up. That is often all a threshold needs. If you pick your auto-route cutoff by measuring the error rate above it on your own traffic, the numbers could be off by 20 points and the rule would still do what you measured.

The second, calibration, is about the numbers themselves, across the range of probabilities the model actually produces. The two come apart easily. If everything labeled 0.9 is right 70 percent of the time and everything labeled 0.7 is right 50 percent of the time, the ordering is perfect and the numbers are 20 points overconfident.

Calibration matters once the number is read as a probability. A threshold chosen because 0.9 sounds like nine in ten delivers three errors in ten at that level from that model. A threshold measured last quarter and carried to new traffic assumes the old relationship between score and risk still holds, and, if the score is also read as a probability, that its calibration carried over too.

Combining probabilities leans on them as well, and takes more than calibration: each number has to be calibrated for its own event, and the combination rule has to match how those events depend on each other. That matters here because TypeSafe recommends asking many narrow questions and combining the answers in code. A naive weighted score assumes the numbers from different questions are comparable. A fitted combiner can learn around some of the differences, but then it needs its own out-of-sample check. If the two underlying events are independent, two reported probabilities of 0.9 imply a joint 0.81. If answers at that level are each correct only 80 percent of the time and their errors are independent, the joint correctness rate is 0.64. When miscalibration is uneven across the range, combined scores can even come out in the wrong order.

Here is the reason for the title. Calibration is not a property of a model by itself. It is a property of a particular configuration, meaning the model version, how state is constructed, the question, and the option set, under a particular input distribution. Change any of them and the claim has to be checked again. Ordering can break under the same shift, and neither failure announces itself, because nothing about a wrong number looks wrong.

What the evidence says so far

TypeSafe’s launch benchmark cannot settle this. It compares every model against reference probabilities averaged from GPT-6 Astra and Fable 5.1 instead of against ground truth, and TypeSafe notes the resulting bias and that its own team built the workflows. As Samuel Sacco’s evidence ledger and others pointed out, that measures agreement with other models, not accuracy against outcomes. If Jev matched the reference exactly, it would be exactly as calibrated as the reference ensemble, whose calibration the launch post does not report.

TypeSafe’s own guidance does not claim otherwise. Its official skill for coding agents says typed output guarantees the interface, not truth, and that the models should be validated in the target domain. The documentation says to test thresholds by plotting confidence against accuracy on your own data. What it does not spell out is why the launch numbers cannot stand in for that test, how differently the answer comes out from one question to the next, and which changes quietly undo it.

Independent testers started running the test within a week, and they did not reach the same verdict.

Mike Moore’s 60-case benchmark on agent tool-call risk asked Jev to classify tool calls as read-only, destructive, privileged, or exfiltration and found 91.7 percent accuracy. All 40 answers with a confidence field of exactly 1.000 were correct, and he read that as calibration holding. Given TypeSafe’s description of that field as a concentration statistic, the result speaks more directly to whether the field ranked risk usefully on his task, at least at the very top of its range, than to probability calibration in the usual sense. A re-analysis of his committed data in Sacco’s repository adds that at 60 decisions and 10 bins, even a perfectly calibrated predictor would score an ECE of about 0.045. Moore’s reported figures, 0.0712 for jev-latest and 0.0505 for jev-preview, sit only 1.6 and 1.1 times that floor, so they could not have settled a calibration question either way. Moore flagged a related caveat himself: 50 of the 60 predictions sat in the top bin.

Sacco then ran a controlled 800-item difficulty gradient on jev-1.13.0. It is synthetic, generated from a 20-pair template pool, and built entirely from yes-or-no questions, which makes it a tool for controlled calibration analysis, not an estimate of production accuracy. A regex on the words that decide each answer already scores 69.5 to 83.0 percent across its tiers. Within that setup, ECE ran 2.1 to 2.5 times its simulated finite-sample baseline in every tier, including one at 97.5 percent accuracy, with probabilities compressed toward the middle: low ones overstated, high ones understated. Because high probabilities were understated, no error was observed at or above 0.9, at 21.5 to 32.5 percent coverage. That slice held only 43 to 65 cases per tier, small enough that zero observed errors should not be read as zero underlying risk. On a separate phishing benchmark whose errors ran the other way, the same 0.9 cutoff, applied there to verdict confidence, was right only 73.9 percent of the time, about one wrong answer in four. Sacco’s summary is that Jev returns a well-behaved monotone score with a stable distortion, not a probability.

A third test, the synthetic-ticket study, targeted the case that matters most for routing. Its authors generated support tickets by script on the day of the run, so none could have appeared in training data, and asked three questions about each of 300 of them. On a four-way queue choice, Jev was 89 percent accurate and slightly overconfident, and 12 of its 33 wrong answers still came back at 1.00. On a yes-or-no question about whether the customer was angry, it was 91.7 percent accurate and underconfident. The study later revised the size of its fitted temperatures, which depended on how exact zeros were handled, but the directions held, and its authors now advise reading the sign, whether a question’s probabilities need softening or sharpening, rather than the magnitude. On three public benchmarks, where training overlap cannot be ruled out, aggregate ECE sat at 1.0 to 1.7 times the corresponding finite-sample baselines.

The third question, a score over four priority levels, matters most. Priority depended on a bump for gold and enterprise customers, a rule the ticket never states even though the customer’s tier appears in it, so no model could fully recover the label from the input. Jev was right 44.7 percent of the time, against 25 percent by chance, at an average stated probability of 0.74, nearly 30 points above its accuracy.

These results disagree on the surface, and part of the disagreement is instructive. The synthetic-ticket study offers a public-benchmark miss, 1.00 on a wrong answer and 0.00 on the right one, as a counterexample to a “no misses at confidence 1.000” observation reported elsewhere, which matches Moore’s forty-for-forty. That item alone cannot settle it, because the two read different numbers. Moore’s figures are for the confidence field, and this miss had a top probability of 1.00 but a confidence field of 0.99, with values quantized to two decimals. The study’s raw results do settle it. On its unseen tickets, 239 queue answers came back with a confidence field of exactly 1, and 12 of them were wrong. Which number a policy reads changes what “never wrong at the top” means, and on a different distribution neither reading held.

Beyond that, each study is right about its own distribution. Across these evaluations, which used different endpoints and aliases, Jev’s aggregate ECE on public benchmarks stayed close to its finite-sample baselines, at most 1.7 times. Its confidence field worked as a high-end risk-ranking signal on one agent task. Its yes-or-no probabilities behaved as a monotone score with a stable distortion on a synthetic gradient, and its answers ran overconfident or underconfident by question type on unseen tickets. The synthetic-ticket study’s advice sums it up: calibrate per question, not per model.

The priority case matters because real routing, approval, and escalation rules can depend on organizational policy that is absent from the raw input. No vendor can pre-establish calibration against a policy absent from both its evaluation and the state your application sends. That takes your traffic, your option sets, and your outcomes, and the strongest operational labels often arrive only after the system has acted.

None of this makes Jev a bad model. Several tests found it strong, and the narrow guarantee held where it was measured. What the independent results illustrate is that a calibration result travels with its distribution. The same holds for any model’s probabilities, including one your own team fine-tuned.

A decision evidence check

Since the evidence has to be produced locally, what helps is a procedure. It answers two questions: how much the system gets wrong where it acts, and, when a policy reads probabilities, whether the numbers mean what they say. If a policy reads per-option probabilities as probabilities, combines them, or carries a probability threshold to new traffic, run all of it. If it only thresholds, whether on a probability or on Jev’s confidence field, steps 1, 2, 6, 7, and 8 are the work, and they have to be re-run whenever the traffic changes.

1. Collect labeled decisions from the traffic you serve. Audit a random sample across automated and reviewed decisions, or the labels inherit the routing policy’s selection bias. Decide what counts as correct before you start, and remember that labels produced by other models measure agreement again.
2. Decide which number the policy reads, and split the sample. If it reads a per-option probability, steps 3 to 5 apply. If it reads Jev’s confidence field, skip them. That value summarizes concentration, which is not the probability of being correct, and TypeSafe’s guidance says that anyone with “a specific statistical algorithm in mind” should probably use the probabilities instead. Judge it by whether it ranks risk usefully, through the operating points in step 6. Either way, fit any threshold, temperature, or calibration map on one split and evaluate it on the other.
3. For probabilities, bin and compare mean predicted probability with observed accuracy. If downstream code uses the full distribution and not only the winning option, evaluate the full distribution, since top-label calibration alone does not cover it.
4. Plot the reliability diagram. A fitted temperature summarizes global over- or underconfidence, but it is only cleanly defined when probabilities avoid exact 0 and 1, and it cannot show whether the direction flips across the range. The curve can.
5. Report ECE against a simulated finite-sample baseline, next to a proper scoring rule. The baseline is the ECE a perfectly calibrated predictor would still show at your sample size and binning, so estimate it for your own data. Treat ECE as a tripwire. Brier score is the simpler companion. If you report log loss, state how exact 0 and 1 probabilities are handled, because Jev’s quantization makes that choice consequential.
6. Measure the operating point directly. For each automation threshold, whichever number it reads, report coverage and selective risk. Under zero-one loss, selective risk is simply the error rate among decisions the system acts on. Where mistakes carry different costs, use the task’s actual loss. Report counts and an interval, because twenty of twenty correct is not two thousand of two thousand. A Wilson interval assumes independent decisions, so if they cluster by customer, incident, or template, use a cluster-aware bootstrap. Treat exact 1.00s separately: once rounding has merged different scores into 1.00, temperature scaling cannot separate them, though a binned or isotonic map can still give that bucket its observed rate.
7. Repeat by slice and by question type. On the same inputs, the synthetic-ticket study found choice and score answers overconfident and yes-or-no answers underconfident.
8. Record the configuration that produced the result. The date, the traffic scope, the model version or alias and whether it was pinned, the state-template version, the question text, the option set, the API or gateway path, and the threshold under evaluation. Without them, a result describes one sample at one moment.

Step 6 is the one I would fight for. A system can look well calibrated in aggregate and still make too many mistakes where it stops asking humans, and a poorly calibrated score can support a sound operating point if its ordering is good and the risk there is measured. When traffic changes, re-measure ranking, coverage, and selective risk, because the relationship between score and risk may have moved. For a score that is not a probability, the question is only whether that relationship transfers. Calibration is an additional requirement when the number itself is read as a probability. Measure where the policy changes.

I am not giving you a threshold, because the tolerable risk depends on what a wrong automatic decision costs. What I would ask is that the risk budget be written down first: the maximum loss you will accept among automated decisions, plus a separate calibration bound if the score is read as a probability. Then choose the threshold on one split so that the upper confidence bound on selective risk, not only the point estimate, stays inside that budget, and report how the locked threshold performs on the untouched split. A budget chosen after seeing the results is only a description of what you got.

Policy, and what breaks it

If the score ranks operational risk monotonically on your task, the policy layer can be simple. Above the automation threshold, route automatically. In a review band, suggest an owner and wait for confirmation. Below it, triage. TypeSafe’s documentation offers a similar starting pattern of three confidence ranges (act automatically, proceed with caution, do not act), with boundaries that depend on the stakes.

What gets skipped is what invalidates the evidence. A model version change is the obvious one. The others arrive unmarked: a large customer shifts the traffic mix, an upstream router sends a different population, someone rewrites the state template or a question’s criteria, a fifth option redistributes probability mass, or a threshold moves into a region nobody measured. None of them changes the model. It is the pattern from Why a Better LLM Can Make Your AI Product Worse: the arrangement around a model is tuned against behavior nobody wrote down, and the calibration result is the part most likely to be treated as permanent, because it arrived as a number.

Where this argument breaks

Verification costs something. If outcomes take a 30-day support cycle to learn, any evidence arrives at least a month late and may already be stale, and a conservative threshold with a wider review band is more honest than a claim you cannot refresh. The cost can also be modest. In the current jev-exploration analysis, transferring the fitted slope and refitting the intercept on 50 labels cut held-out ECE by 62 percent. Its author treats 50 labels as a floor, not a recommendation, since below 30 labels the correction could leave calibration up to four times worse than doing nothing. That was one synthetic domain, on yes-or-no questions, with one model version.

Abstention costs something too. A system that abstains whenever a score is imperfect can be worse than one that acts and fixes the occasional mistake, especially where correction is cheap. The risk-coverage tradeoff is a tradeoff, not a direction.

The independent tests are small, and they are not mine. The operational tests are modest: 60 cases in Moore’s benchmark, 800 in Sacco’s gradient, and 300 per question type in the synthetic-ticket study, run on different endpoints. Public-benchmark slices go up to 2,000 items, but they answer a different question. Sacco’s gradient is synthetic and yes-or-no only, so it does not establish the same curve for choice or score questions. Jev may have changed since any of them ran. As Turing Post notes, not enough is public to separate the contribution of RLCD, TypeSafe’s reinforcement learning for calibrated decisions, from architecture, data, or serving, and nothing here is a claim about Jev’s internals.

The disclosure could have gone the other way. This argument is concrete only because TypeSafe published its caveats and independent testers published raw responses and checked each other’s numbers.

Make recalibration routine

It is easy to run a check like this once and never again. Attach it instead to the things that invalidate it. Shipping a model version, editing options or templates, rerouting traffic, and moving thresholds are all deliberate, so each can reopen the check the way a schema change reopens a migration review. Hold the model version to the firmest rule: a new version always reopens the full check. In the jev-exploration follow-up, the correction’s slope held within a narrow band across difficulty tiers while its intercept moved with each tier’s base rate. A first cross-domain test put none of three new-domain slopes inside that band, and survival across a retrain is untested.

For drift nobody decides, put a date on the evidence. When the date passes without a fresh check, and an unnoticed error would cost more than a review, narrow automation until the evidence is refreshed, so that stale evidence fails toward a human instead of away from one.

A model can ship probabilities. It cannot ship evidence that they are calibrated on your traffic, or that your thresholds hold up there.

That evidence belongs to the system operating the model. And it expires.

In the terms of the Quality Locus, calibration is not a property of the weights alone. It lives in the evaluation and runtime surfaces, which measure confidence on your own traffic and decide which threshold the system acts on.

---

The distinction between confidence and calibration, and why a calibration result carries its distribution with it, is the subject of Chapter 12, “Knowing When You Don’t Know,” in Beyond the Model.

Sources

TypeSafe AI. “Introducing System One Models and Jev.” September 15, 2026. https://typesafe.ai/blog/introducing-system-one-models-and-jev

TypeSafe AI. Documentation pages “System One,” “Confidence,” “How to build with TypeSafe,” and “AI primer.” Accessed September 25, 2026. https://docs.typesafe.ai

TypeSafe AI. “typesafe-ai” agent skill (SKILL.md), commit 65a39f3. https://github.com/typesafe-ai/skills/blob/65a39f393687675ce170e6094757de20370365b9/skills/typesafe-ai/SKILL.md

Moore, M. “I Benchmarked Jev on Agent Tool-Call Risk. Calibration Held.” Web of Mike, September 19, 2026. Run on September 17, 2026, against jev-latest and jev-preview. https://webofmike.com/jev-benchmark/ Repository at commit 42a9123: https://github.com/themsquared/jev-benchmark/tree/42a9123a4ef42488aa775effdecb2f34a0d66a5a

Sacco, S. “Jev: an evidence ledger.” GitHub, commit 76c793c, accessed September 25, 2026. Difficulty gradient run on jev-1.13.0. https://github.com/SamuelSacco/jev-exploration/tree/76c793c5a0538052dbaf60134b1661904bb4a6aa Re-analysis in pull request #8: https://github.com/SamuelSacco/jev-exploration/pull/8

scienthoon. “Does Jev know when it doesn’t know? Calibration of TypeSafe’s Jev on an unseen rule-based task.” GitHub, commit 914d87a, including the correction dated September 22, 2026. Run on September 19, 2026, through Vercel AI Gateway model id typesafe-ai/jev. The underlying model version is not exposed. Counts of answers at a confidence field of exactly 1 are from the raw files in its results folder. https://github.com/scienthoon/jev-ood-calibration/tree/914d87ab16517f14e833cff63d73b51235b14bd6

Se, K. “Guide: Jev, RLCD, and the Reinvention of the AI Classifier.” Turing Post, September 19, 2026. https://www.turingpost.com/p/what-is-jev-rlcd