Jev: The Good, the Bad, and the Ugly. What Happened When We Asked the Same Questions Ten Times.

We sent one model the same question ten times. Same question text, same input bytes, same model identifier. It answered “no” eight times and “yes” twice, and reported at least 85% confidence on every one of those ten answers, including both incorrect answers.

That is the finding this piece is built around, and it is not the one we set out to measure.

We were running a head-to-head between two very different engines on the questions FlowX.AI actually asks in production. One is a chat model, GPT-5.6 Luna. The other is a typed classifier: TypeSafe’s Jev, a model that cannot generate text at all. You hand it a state and a map of typed questions, and it answers all of them in one forward pass against option sets you supply. It is structurally incapable of returning an answer that was not on the list.

The accuracy comparison landed roughly where we expected. The repeatability test did not, and it reframed what the accuracy comparison was worth. There is a good story here, a bad one, and an ugly one, and the ugly one is the useful one.

What we measured, and why

FlowX.AI runs typed classification in four places today. Observatory uses it to find the failure that does not look like one: the agent run that reports SUCCESS, throws no error, and is simply wrong. Agent Builder offers it as an alternative decision engine on orchestrator and evaluator nodes. Our evaluation service scores security rubrics with it alongside the existing LLM judges. And a document classifier uses it to second-guess an LLM judge’s self-reported confidence, which is what currently decides whether a page is trustworthy enough to enter a training corpus.

Those are four different jobs, and a generic classification benchmark would have told us nothing about any of them. So we built the suite out of the questions themselves.

The question text is copied verbatim from the deployed source. Not paraphrased, not simplified. This matters more than it sounds: an early draft used short rubric labels (severe, borderline, clean) in place of the real multi-sentence criteria, and the classifier scored two points lower. It reads criteria text as part of the question. Paraphrasing the prompt changes the measurement.

Both engines get one call per state. The classifier answers a whole question map in one pass. The chat model gets one structured-output call returning one object with all the answers. Charging it per question would have manufactured a cost gap nobody would actually pay.

The chat model gets a JSON Schema enum on every closed set, which is the strongest constraint that API offers. Both engines returned zero invalid options across the whole suite. Worth stating plainly, because “it cannot invent an option” is the advantage most often claimed for typed classifiers, and on this suite it never became one.

Nothing is retried, and the labels are hand-assigned by one person and deliberately unambiguous. Genuinely contested states are marked as such and excluded from accuracy. This measures agreement with an obvious reading, which is a real but modest thing to measure.

That gave us 22 labeled states and 106 judgments across three families: semantic trace failures, security rubric scoring, and orchestrator routing.

The good

Start with the table, including the row that does not flatter the classifier.

The chat model wins the headline: 84.7% against 76.5%. Read the next three rows together, though, and the aggregate turns out to be the wrong summary. On trace failure detection and orchestrator routing, the two engines tie exactly: 50 out of 58 on trace failures, 5 out of 5 on routing. Every point of the 8.2-percentage-point gap comes from one specific job, which gets its own section below.

The last row is the one that survives everything else in this article. When the classifier is wrong, it reports a mean confidence of 0.555. When the chat model is wrong, it reports 0.965. A wrong answer delivered at 0.97 confidence is the expensive kind, because high confidence alone does not flag it for review.

Then there is speed, which is where the two stop being comparable things with different scores and start being different categories of component.

Median latency is 0.77s against 1.98s, a 2.6x difference. The tail matters more: a 95th percentile of 1.13s against 3.38s. The classifier’s slowest call in the whole run, 1.19 seconds, finished faster than the chat model’s median. For a detector meant to run over every trace in a time window, that difference can materially affect how often a scan can run.

Cost separates them further, and not for the reason most people assume.

The classifier read more input than the chat model over the same suite: 17,130 tokens against 14,169, about 21% more. It still billed a tenth as much. Two factors explain that. Its input rate is $0.042 per million tokens against $0.20. And it bills nothing for output, while output is 60% of the chat model’s bill on this workload. That output charge alone, $0.196 per 1,000 calls, is six times the classifier’s entire invoice.

The ratio is a property of this workload, not a universal constant. These are short-context calls at the rates published on the vendors’ own pricing pages when we ran the suite, and list prices move. Check the current rates before carrying 9.9x into a business case.

The bad

All 8.2 percentage points of the accuracy gap come from graded rubric scoring, and the raw values reveal a pattern worth investigating.

One of the security states contains a genuine personal-data leak and nothing else wrong. Five rubric questions share one forward pass: is it toxic; is it a refusal; does it leak personal data; is it a prompt injection; and is it a jailbreak? On this platform, and throughout this section, a high score means the check passed. Four of the five answers should sit at 4. One should sit at 0.

The classifier gets the actual question right. It returns 0.01 for personal-data leakage, close to the expected score of 0, at 0.99 confidence. The other four scores are also low: 1.12 for toxicity, 0.31 for refusal, 2.85 for prompt injection, 1.29 for jailbreak. All four should read 4. Nothing in that state is toxic, and nothing in it is a jailbreak.

The chat model, asked the same five questions in the same single call, got four of the five right. Five answers from one forward pass should not be assumed to be independent. Something being badly wrong appears to bleed across the whole question map.

This has a direct operational consequence, and it is why our evaluation service runs the classifier in shadow mode with its own columns instead of switching any metric over. Move one rubric dimension onto an engine that behaves this way and you risk introducing errors on four dimensions you did not intend to touch. The shadow columns exist to measure exactly that at production scale before anything changes.

This result does not establish how the engine behaves on every task. Question maps about different properties, such as “which class is this document?” alongside “is this page legible at all?”, need their own checks for correlated errors.

The ugly

Everything above is a single-shot measurement. Both engines report a confidence number with every answer, and the natural reading of a benchmark table is that those numbers mean something stable. So we tested it: four states, ten times each, to both engines. Byte-identical input, same model identifier, same question text. That is 40 calls per engine, 80 in total, producing 210 question-level answers per engine: 21 questions repeated ten times each.

Neither engine returned the same number every time.

The classifier moved by up to 0.12 on a zero-to-one scale, and only 4 of its 21 questions returned one identical value across all ten calls. Our own code comment claimed these values “differ in the third decimal.” The observed spread was as large as 0.12. We have corrected the comment.

The chat model was, on average, slightly steadier in its numbers: a mean spread of 0.027 against the classifier’s 0.035. It was also the only one of the two that changed its actual answer.

The figure is the argument. On this question the correct answer is no. The chat model said no eight times and yes twice, at 0.98 and 0.95 confidence on the two occasions it was wrong, and it never once reported below 0.85. Its confidence number does not distinguish the answer it kept from the answer it abandoned. It was confident both ways.

The classifier’s values stayed within a range of 0.02 across all ten calls, and its verdict was wrong on every one of them. It is the more reproducible engine here, and on this question it is reliably reproducing a mistake.

That is the whole point, and it is worth being blunt, because the intuitive lesson from the top lane is “prefer the stable engine,” and that lesson is wrong. Reproducibility is not correctness. A dependably wrong answer is still wrong. A consistently reproduced mistake is easier to find, characterize, and correct. You can write a regression test for it. An answer that is right 80% of the time needs repeated trials to assess reliably: a single test can pass on Tuesday and fail on Wednesday with nothing else having changed.

So we do not treat either engine’s confidence as a measurement. We treat it as a number that came out of a model, which is what it is. In Observatory, choice confidence is recorded for display and never used as a threshold, because we measured it at 1.00 on a clear-cut contradiction and 0.99 on a genuinely ambiguous trace, so it does not separate the two cases at all.

What determinism buys you

Here is the number that reconciles all of this.

Across those 210 question-level answers per engine, the classifier’s values wandered by up to 0.12 and produced zero changes of verdict. The chat model’s values wandered slightly less, and produced six.

The values move. They simply do not move anywhere that matters, because the threshold sits far from where they live. Three questions answer around 0.95 to 0.99, three answer between 0.02 and 0.38, and the gate is at 0.75 with nothing near it. That margin kept the fired rules stable across these repeated calls. Rounding to two decimals affects the display; it does not guarantee a stable verdict on a future scan.

The chat model’s six verdict changes tell a different story. On one rubric question, its ten reported confidence values were 0.99, 0.99, 0.98, 0.98, 0.99, 0.99, 0.99, 0.98, 0.99, 0.99, a total spread of one hundredth of a point, and the reported rubric level still changed. The confidence barely moved, but the answer did. Those confidence values do not tell us where the model’s decision boundary lies or why the answer changed.

For the classifier, the margin between the observed values and the threshold explains the stable verdicts in this sample. This illustrates the architectural argument our RAILS paper makes about agent architecture generally. Its second section is explicit that determinism “does not require that a generative node emit identical tokens every run; that is neither achievable nor necessary.” What it requires is that the parts correctness depends on are fixed: the control flow, the typed contracts, the bounded side effects. Generation is allowed to vary, inside a node whose output is then validated or consumed deterministically.

A classifier is that thesis at a smaller scale. The model’s output wobbles; the boundary it is compared against does not; the fired rule downstream remains stable as long as the output stays on the same side of that boundary. Determinism is an architectural property you engineer at the boundary, and not a model property you hope for.

It is worth being careful about the claim. We are not saying FlowX.AI invented typed model outputs, and the RAILS paper’s own bibliography cites the guided-generation literature that predates it. What we will say is that two teams reached the same constraint from opposite directions. We came at it from orchestration: make a system reliable by confining where generation is permitted to matter. TypeSafe came at it from model architecture: build something that structurally cannot answer outside the option set it was handed. The convergence is the interesting part, and a stronger observation than a priority claim would be.

Where we think it works

Typed classification is behind a flag in every service that has it, off by default, and every service falls back to “no verdict” instead of an error when the key is missing, the request times out, or the response cannot be read. A component behind a flag must never be able to fail the thing it is observing.

Observatory runs it as a detector, one of the two families where the engines tied. Our evaluation service and the document classifier both run it in shadow mode, writing to their own columns beside the existing LLM judges, because the cross-contamination result says plainly that we do not yet know enough to switch a metric over. Agent Builder offers it as an opt-in decision engine per node, with the LLM path still the default.

The document classifier is the case we are most interested in, and it is not about accuracy at all. What currently decides whether a page is good enough to enter a training corpus is an LLM judge’s self-reported confidence, and the measurement above is exactly why that is uncomfortable. Replacing a number a model invents about itself with a real distribution over a closed label set is a calibration argument. Whether it holds is what the shadow run is there to find out.

And the honest limits. A suite of 22 states is enough to show a roughly 10x cost gap on this workload and to expose a specific, reproducible failure mode. It is not enough to confidently rank two engines that finish about 8 percentage points apart, and an earlier identical run gave the chat model one more correct answer on trace failures than this one did, which is the single-shot jitter showing up in our own headline table. One labeler, no adjudication. Synthetic states, not sampled production traces. The harness is committed alongside the services it tests, so the next run starts from the same rules.

Glossary