Observatory 2.0 · live demo environment
Most AI observability tells you what happened. This tells you what recurs.
Four surfaces, one argument: see what the agents did, prove it is controlled, prove it pays, and let it improve itself. Every number on this page was read live from the demo environment — none of it is illustrative.
Observatory Hub
“We have no idea what our agents are actually doing.”
Governance Hub
“We can’t put this in production — we can’t prove it’s controlled.”
ROI Hub
“Finance wants to know what we got for the spend.”
Self-Improvement
“It’ll be stale in six months and nobody will own it.”
See everything. Then be told what matters.
37,204 runs in the demo org alone, as hierarchical traces from agent to chain to tool to LLM call, with latency percentiles, cost per model and error rate. That part everyone has. What comes next does not exist elsewhere.
Recurring failures become one actionable issue
A successful API response can still contain a wrong business decision, and error rates alone miss that entirely. Trace Insights reads the signals that do carry it — evaluation scores, thumbs-down, discards, retry loops, behavioural shape — then collapses the runs that share a cause into a single issue with an identity that survives the next scan.
open recurring issues
runs affected
of them silent
alerts that fired
The number that should worry you
89% silent. A silent failure returned success end to end and still did the wrong thing — no exception, no error status, nothing on the error-rate chart, nothing for an alert threshold to catch. That signal does not live in any single trace. It lives in the shape of many.
34 claims approved. All successful. All ungrounded.
Deterministic first. LLM last, and gated.
Detection and clustering make no model calls at all. Seven rules run over root runs; a structured signature is rendered; runs group on an exact key and only then refine. A label — the one paid step — is bought once per new or materially changed cluster, never per run.
| Scan | Flagged runs | Clusters | LLM calls | Cost |
|---|---|---|---|---|
| First, labelled | 1,134 | 17 | 17 | $0.0095 |
| Rescan, nothing changed | 1,134 | 17 | 0 | $0.0000 |
| Scheduled overnight ×3 | 1,116–1,128 | 17 | 0 | $0.0000 |
Under a cent for 1,134 flagged runs, then free. The feature that finds expensive problems must not become one.
benefit
Engineers investigate a shared failure once — see how many runs it affects, and prioritise the issue — instead of reviewing logs one run at a time.
Prove it is controlled — to someone who can say no.
The Readiness Index answers what no wall of metrics can: is this workflow fit to run unattended? Four weighted inputs, a deterministic score, and a control gate that outranks the average.
compute_score is deterministic — no LLM sits in the scoring path. The same inputs and the same scoring-model version always produce the same number, which is what makes a score defensible six months later rather than merely archived. When ROI is insufficient_data its weight redistributes and the score is flagged provisional, so a missing input never silently inflates a result.
A control failure overrides a good average
This is the part that matters in a regulated deployment. Look at the middle bar: it scores 82 — higher than the workflow next to it that reads Amber — and it still reads Red, because it is an EU AI Act high-risk workflow whose audit-trace completeness sits below 90%.
regulated-v1| Workflow | AT | ATC | RR | ROI | Composite | Band |
|---|---|---|---|---|---|---|
| Claims Auto-Approval | 100 | 98 | 100 | 100 | 99 | green |
| Mortgage Underwriting | 80 | 82 | 85 | 80 | 82 | red |
| Claims Exception Handler | 90 | 92 | 65 | 60 | 80 | amber |
The four blocking gates
high_risk_low_atc — high-risk with ATC below 90%. autonomous_no_kill_switch — autonomy L3+ with no kill switch. high_risk_l4_no_reversibility — high-risk L4/L5 with no reversibility. high_reversal_irreversible — over 10% reversal on irreversible actions.
Regenerate it in six months. Identical bytes.
An evidence pack is built as a pure function of read-model state — it re-runs the deterministic core rather than reading a stored score, and omits every volatile field. Immutable versioned profiles mean a historical score is reproducible, not just archived. JSON or PDF, clause-mapped to SR 11-7 and the EU AI Act.
One more thing only a join can see: the Governance Hub surfaces high-risk uncovered — a project classified high-risk with no active policy pack. Either fact alone is unremarkable. Together they are a gap, and no single page can find it.
benefit
Risk owners get a specific remediation target and reproducible review evidence. A strong ROI score cannot hide weak controls, and the institution keeps the sign-off decision.
Prove it pays — to someone who audits numbers for a living.
Counting executions tells finance how much AI ran. It does not establish what the work was worth. Every value claim here carries an evidence grade.
execution estimates
calibrated hours saved
attributed value
net ROI
The asymmetry, at one scale
The reason the ROI conversation is short: draw what it produced and what it cost on the same axis, and the cost stops being visible. That is not a rhetorical trick — it is the same scale, and the sliver is the whole spend.
Anyone can multiply hours by a rate. Why would you believe it?
Because two independent models are computed and reconciled rather than one being asserted. A flat baseline is declared manual effort per agent. A counterfactual estimator has an LLM judge how long a competent human would have taken, then calibrates that against the org’s own historical samples. The two reconcile against each other, and where they disagree, that is the conversation.
Sensitivity, not a point estimate
Monte Carlo across 10,000 simulations, and a five-year TCO against the alternatives.
Reconciliation per agent
Where baseline and estimator disagree, that disagreement is surfaced as the thing to discuss.
An honoured opt-out
Per-agent, operational opt-out so a team can say “this should not count” — and it stops counting, platform-wide. An excluded pair also stops being a Trace Insights candidate. One decision, respected everywhere.
benefit
Finance can challenge the inputs, see how much value is estimated versus evidenced, and compare investment options on the same basis. Returned staff capacity becomes cash savings only if operating spend actually falls — and the Hub says which is which.
The loop that closes it — governed by the same machinery.
Everything else here is about seeing and proving. This closes the loop: the system proposes changes to an agent, tests them against real evaluation data, and surfaces only what survives. It is not the agent editing production on its own.
Three properties make this safe to run where it matters.
A canary, not a deploy
A mutation only ever sees a traffic slice, under a budget cap, until it is promoted — so a bad mutation cannot run away with spend.
Significance, not vibes
Auto-promotion requires statistical significance. Without it, promotion stays a human action.
The same measuring stick
Mutations are scored on the same evaluation dimensions as everything else, so an “improvement” cannot be an improvement on a private metric. Every candidate’s scores stay in the matrix — winners and losers alike.
benefit
Teams compare targeted fixes, retain the reasoning behind each release decision, and reverse a change that performs poorly — so the loop supports ongoing maintenance as workflows and data change.
Four surfaces, three different questions about the same trace.
They are not four products, and the joins are the argument:
The loop actually closes
A promoted mutation that misbehaves appears in Trace Insights as a new growing cluster. ARI consumes ROI as a scored dimension, so a workflow that does not pay cannot score green on value.
One decision, respected everywhere
An operation opted out of ROI also stops being flagged as a failure candidate. And every export is an open format — W3C PROV-O JSON-LD, byte-stable evidence packs, cluster JSON.
What recurs, whether it is fit to run, what it is worth, and how it improves.
Bring a workflow you cannot yet run unattended — we will score it, show you what recurs across its traces, and hand you the evidence.