Browser Agents Don’t Fail on Capability. They Fail on Consistency.

In one benchmark run, a browser agent approved the requested ticket and a second ticket it had never been asked to approve. In another, it submitted the same claim twice. Both runs were reported as successful.

A check that asked only whether the requested action happened would have missed the failure. Our benchmark checked whether exactly the requested action happened.

We ran nine models across nine browser tasks, five times each: 405 runs in total. Five models passed all 45 of their runs. The results exposed substantial differences in cost, task-specific failures, and the accuracy of reported completion.

What we built, and why we measured it

Most enterprise automation eventually meets a system with no API. A supplier portal, a government filing site, an insurer’s broker extranet, an internal tool whose owner left in 2014. The integration layer stops at the edge of the browser, and the process carries on inside it, done by hand.

FlowX.AI Agent Builder has a browser automation node. Give it a target URL, a task in plain language, and optional form data, and it drives a real Chromium session. It uses browser-use, the open-source browser-agent library, wrapped in the platform’s execution, logging, and document-storage layers. Each step is logged and its screenshot retained.

The browser automation node ships with FlowX.AI 6, in preview now and generally available this fall. Building it took a few weeks. Establishing when to trust it takes more than a demo: will it complete the process correctly across repeated runs, and can we detect when it does not?

How we measured

The benchmark covers bounded tasks on a purpose-built site, with five repetitions per model-task pair. It measures performance on these fixtures; it does not establish reliability across thousands of production executions or on the open web.

We verified outcomes independently. We checked recorded submissions and state changes, or compared returned answers against the expected result. The table keeps the run’s reported success status in a separate “Claimed” column. That status is not our completion criterion. We use overconfidence rate for the share of all runs that reported success but failed verification; the difference between aggregate claimed and verified rates is a separate, signed gap. A claimed rate exists only where the run produced a schema-valid final report: a model that cannot emit one has no claimed rate, and we report that rather than scoring the silence as a denial.

We hosted the target ourselves. A control endpoint exposes the fixture site’s recorded state and resets it between runs. This keeps the target and scoring rules stable while agent behavior can vary. Live third-party sites introduce changes outside the benchmark’s control.

We repeated every model-task pair five times. Repetitions expose outcomes that a single attempt can miss. Five runs per task are still a small sample, so we report the counts alongside aggregate rates.

The nine tasks cover a three-page form; a modal and dynamic dropdown; recovery from a validation error; an iframe form; shadow-root controls; a row loaded after four scrolls; a fact carried between pages; a queue whose rows change order; and a wizard with visible dead ends and a hidden valid option.

Five models passed every test

Five models completed all nine tasks, five times each, without an observed failure. That includes the iframe, shadow-root, scrolling, and recovery tasks.

Gemini 3.8 Flash was among them. We had already hardened the suite after an earlier round reached perfect scores, but these tasks still did not distinguish the five models by pass rate. A broader or more demanding suite may do so.

The other four models failed in different ways. Their task-level results help identify where a model needs further testing, a different execution strategy, or a fallback.

The table reports 45 scored runs per model, ordered by cost per run. Costs reflect the token rates used for this benchmark; they are not a forecast of total production operating cost.

^† Haiku 4.5 has no comparable claimed rate. In 42 of its 45 runs the harness never received a schema-valid final report from it, so the run carries no self-assessment either way. The three runs that did report are covered below.

GPT-5.6 Sol and GPT-6 Astra both passed 45/45 runs. Sol cost 42% as much; Astra had lower median and p95 latency. Gemini 3.8 Flash also passed 45/45 at 3.52 cents per run, making Astra 26.5 times as expensive at the rates used. Haiku cost about twice as much as Gemini and passed 60% of its runs. These results support comparing cost and latency among models that meet your own task requirements.

The cost comparison is sensitive to pricing. The Gemini calculation uses promotional rates recorded for the study. If those rates double while token usage and Astra pricing stay unchanged, Gemini’s cost would rise to about 7 cents per run and the gap would narrow to roughly 13×. Check the applicable rates before using this comparison for a deployment decision.

Latency also separates models. Haiku’s median run took 79.9 seconds and its p95 took 208.4; Grok’s were 33.2 and 121.8 seconds. Astra’s p95 was 49.1 seconds and Gemini’s 43.6. Across the field, p95 varied by almost five times. This describes the measured tail, not the maximum duration or an SLA guarantee.

Look at the task-level failures

Aggregate rates hide whether a model struggles with one specific task or varies across several.

Mistral Medium passed 73.3% of its 45 runs. It completed the three-page form, iframe form, and validation-recovery task three times out of five each. Those mixed outcomes show why a single successful attempt is insufficient evidence for deployment.

Grok 4.6 passed 86.7% overall, with five passes on each of seven tasks. All six failures came from two tasks: the rotating queue, with three passes in five, and dead-end recovery, with one in five. The aggregate score conceals those weaknesses.

Task by task, for the four models that failed at least once. Luna’s column is marked: at 41 of 45 it is the strongest of the four, and the cheapest model in the study — but the single task it cannot do is the one Grok passes five times out of five.

The scroll task separated models sharply. Mistral and Haiku scored 0/5; Luna scored 1/5 while passing every run on the other eight tasks. Gemini passed the scroll task at a lower cost than Mistral or Haiku, so the results do not establish a price threshold for this capability. Mistral and Haiku also had mixed outcomes on several other tasks. Test the interactions your own portals require.

Luna, Grok, and Mistral reported success on every run. Their overconfidence rates were 8.9%, 13.3%, and 26.7% respectively. A leaderboard based on reported status would give all three a perfect score.

Haiku 4.5 failed in a different way: it mostly did not report at all. In 42 of its 45 runs the final message did not satisfy the framework’s output schema — the required action field was missing — so no status was recorded. No other model in the field produced a single such run. The three runs where Haiku did return a valid report all claimed success, and all three passed. Its self-assessment is therefore unusable rather than pessimistic, and no calibration gap can be computed from it.

Opus reported failure on one run that passed verification, so reported failure can also conceal completed work — but at one run in 45, not at scale.

Gating on Mistral’s reported status would accept all of its failures as successes: 26.7% of all runs. Gating on Haiku’s status would reject almost everything it completed, because there is usually no status to read. Both need a check on what actually happened before continuing or retrying.

When success includes an extra action

The queue task asks the agent to approve only ticket TCK-3312 while row order rotates on every render. Grok approved two tickets in one run. On the dead-end claim task, it submitted the requested property-damage claim twice. Both runs failed verification.

These are extra actions with potential operational consequences. In a production workflow, duplicate submissions could create duplicate records or downstream work. Similar behavior in a payment workflow could be more serious, although payments were not part of this benchmark.

Checking whether TCK-3312 was approved would have marked the queue run as successful. Detecting the failure required checking that exactly one ticket was approved and that it was the correct one. Success criteria need to cover unwanted side effects as well as the requested outcome.

What this means for governed automation

The results point to four practical controls for browser automation.

Qualify models on your tasks, then compare cost and latency. Five models had identical observed pass rates across a 26.5× cost range. That makes lower-cost candidates worth testing, but it does not guarantee equivalent performance on your processes. Include the portal interactions and failure paths that matter to your operation.

Verify completion against the system of record. Keep reported status as a diagnostic signal. Before advancing a workflow or retrying a failed run, check whether the intended state change occurred and whether anything else changed.

Separate authorization, duplicate prevention, and audit. Human approval can authorize a consequential action. Preventing it from happening twice also requires execution controls, such as a unique operation identifier enforced by the receiving system where supported, plus a check of recorded state before retrying. Approval alone does not prevent duplicate execution.

The FlowX.AI browser node records each step’s action, stated reasoning, and next intended goal, and retains a screenshot. Those artifacts support investigation: what did the agent see and do? They complement outcome verification; they do not replace it.

Repeat tests and exercise recovery paths. Include lazy loading, moving targets, validation errors, and dead ends. Score extra actions explicitly. Increase the sample size and task coverage before treating a promising benchmark result as evidence for a production rollout.

This study covers known tasks with definite completion criteria on a controlled site. Open-ended research, changing third-party pages, authentication failures, and other production conditions need separate evaluation. The finding is narrower and useful: several models handled these browser interactions consistently in our sample, while outcome verification exposed failures that reported status missed. For the open-ended case, browser-use’s own published benchmark runs tasks against the live web and scores them with a model-based judge, reporting its field between roughly 52% and 80%. Both results can hold at once: driving a browser through a known flow is a different problem from working out what to do on an unfamiliar page.

The fixture, task definitions, assertions, and scoring harness are not currently published, so readers cannot independently reproduce these measurements today. The fixture and scoring rules are deterministic; agent execution is not. Costs use the rates recorded for the study, and every one of them was checked against the vendor’s own published pricing page. Two notes on identifiers and billing. The Mistral row ran against a floating -latest alias, which currently resolves to Mistral Medium 3.5, so a later re-run may not test the same model. And xAI applies its long-context tier as a cliff, billing every token in a request at the higher rate once the prompt reaches 200k, so any Grok run that crossed that line is understated here by a clean factor of two and not by a fraction. These limits should travel with any comparison based on the figures.

Glossary