Author classes
Define the class taxonomy — labels, descriptions and positive/negative examples. At least two classes, plus a catch-all for anything that fits neither.
Research · Methodology
How we build& evaluate models.
An LLM labels, humans review, deterministic models train into immutable versions — and nothing goes live until it clears a frozen test set.
human-reviewed labels · frozen test set
The methodology thesis
A model you can't reproduce, measure, or retrainisn't ready for regulated work.
So we build for exactly those three. Labels are earned through human review, not scraped. Models are deterministic and immutably versioned, so a decision can be reproduced. And every version is measured against a held-out set before it is allowed to go live.
Build → evaluate → promote → retrainHow we build
From taxonomy to a promoted version.
A configurator authors the classes; an LLM and a reviewer build the corpus; the platform trains, measures and promotes — then retrains from production.
Label with a human in the loop
An LLM auto-classifies each page; humans review a confidence-ranked queue — low-confidence pages float to the top — behind a full-review gate before anything enters the corpus.
Train an immutable version
Train on the reviewed corpus into a versioned, immutable model. Classifiers are deterministic SPLADE + LightGBM — not an LLM — so the same document always yields the same result.
Evaluate & promote
Score against a frozen test set, clear the promotion gates, and set the active version. Retrain from production traffic as it accrues — without losing the version history.
Model choice
Why a trained model, not a prompt.
For high-volume classification we reach for a deterministic model over an LLM node — and the reasons are the ones regulated work cares about.
Same input, same output
A deterministic classifier returns the same result on the same document every run — reproducible and auditable, which an LLM call is not.
Latency & cost at scale
A CPU-bound SPLADE + LightGBM pipeline is faster and far cheaper than an LLM per page — the difference that makes enterprise volume viable.
Purpose-trained
A model trained on your documents and your taxonomy beats a general model asked to guess the same labels.
The stochastic model still has its place — it does the labeling. It just doesn't make the final call unmeasured.
How we evaluate
Measured before it ships.
Evaluation isn't a step you can skip — it's the gate a version has to pass to become the one in production.
A frozen test set
A stratified 80/20 split moves ~20% of each class into a frozen test pool at entry. Assignments never flip, so the benchmark stays honest.
- 80/20 stratified
- per-class
- never flips
- held-out
Every version, measured
Each trained version ships overall and per-class accuracy, a confusion matrix, and the training distribution — inspect before you promote.
- overall accuracy
- per-class
- confusion matrix
- distribution
Promotion gates
A version can’t go live until it clears the gates — a guard against silently shipping a regressed model.
- ≥2 classes
- ≥10 pages/class
- 2.5× imbalance warn
- promote-to-main
Read more: The technical-paper series → · Safety & governance →
Next
Bring aclassification task.
Bring a document set and its labels. We'll build the taxonomy, label with review, train a version and show you the confusion matrix — on your data.