I raced 34 browser automation tools on the same five-second job

Every browser automation tool has the same marketing line: fast, cheap, reliable, one line to install. None of them are compared on the same job under the same conditions, so "fastest" is usually whatever the author measured on their own laptop.

I wanted the boring, comparable version. So I built web-automation-microbench: one fixed job, one browser situation, one pass test that reads the page independently of whatever the agent claims. Then I ran 34 tools against it and put them on one leaderboard. The full leaderboard is here.

One job, one honest pass test

The task is deliberately small: on the public TodoMVC demo app, add two to-dos ("Email supplier", "Review invoice"), tick off only the first, switch to the Active filter, and confirm only "Review invoice" is left with "1 item left".

Two rules keep it honest:

1. Pass or fail is decided by code, not by the agent. A separate checker reads the page and the app's saved data. If a tool says "done" but the todo was never committed, it fails. That mattered more than I expected.
2. Failed runs are kept. Nothing is retried away to make a median look better.

The stopwatch covers only the model-plus-browser work — first model call to "done" — not browser startup or the checking step.

What actually won

Thin harnesses that let the model issue one native browser command per step won on both speed and cost. The top of the table is extension-backed CLIs finishing in about four seconds for half a cent:

The pattern is not "which company", it is "how much ceremony sits between the model and the browser". Anything that runs its own heavyweight agent loop — BrowserCode, notte, skyvern, midscene — landed at 33 to 187 seconds for no accuracy gain. midscene burned roughly 444k input tokens per run doing it. Once you see that on one screen, the choice gets easier.

The part that surprised me

A leaderboard sorted by time tells you almost nothing about whether a tool can do real work. So I scored the same harnesses a second time on 11 real tasks pulled from my own recorded sessions — tag inspection, script inventory, SEO audits, crawlability checks, and consent-to-add-to-cart journeys on live retail sites.

The two rankings disagree, hard. BrowserSkill is the fastest tool on the microbenchmark and among the weakest on real work. browser-relay is a hair slower on the toy task and the strongest on real work. If I had only published the speed table, I would have recommended the wrong tool.

The difficulty split is clean, too: one-shot "inspect the live page and report" audits were attempted 10 times each across the field and typically passed 8 or 9 of those attempts — one SEO variant excepted, where models over-report nested JSON-LD types. The multi-step journeys — consent, find the product, add to cart, verify the tags — passed 0 to 2 times out of 10. Complexity, not harness quality, is what broke them.

The newest tool did not break the rule

The most recent addition is the most interesting, because it was supposed to break the pattern above. Browser Use Pi does not issue one browser command per model step — it runs its own agent loop, writes JavaScript into a persistent V8 REPL, and drives Chrome through raw CDP, building its own helpers as it goes. That is exactly the "heavier architecture" the speed table punishes, and the hypothesis was that heavier would win once the work got long enough to need composing.

It did not, on this evidence. The speed table punishes it as expected: 53.5 seconds median over ten reps against the leader's four, with three of those ten ending in a false "done" that never left the default filter. And on the real-work corpus its 4 passes out of 11 are a strict subset of what browser-relay — a thin CLI thirteen times faster on the toy job — already gets.

I want to be precise about one thing, because I got it wrong in my own notes at first. It does pass the CHANEL product-page tag inspection, the task where the site's anti-bot behaviour stops most harnesses. That is not unique to it: browser-relay passes the same task two times out of three, and the task sits at three passes out of ten across the whole field. It is a genuinely hard task and the new tool handles it, but it does not buy you a capability the cheaper tool lacks.

So the honest reading is narrower and less exciting than I expected: the heavyweight architecture is a real capability that this task set does not reward. Its round-trip count is low — 3 to 6 model calls on the one-shot audits, where a thin CLI needs 8 to 14 — so the code batching does exist and does something. It just does not convert into speed or reliability yet. The architecture permits bigger composed actions; the model largely used it as a slower one-command-per-step loop. I do not have evidence that code-batching wins on long authenticated workflows, and this run is mild evidence that it does not.

The newest own-loop contender lost outright

Round 8 added adaptive-ui-runtime — an own-loop runtime with its own plan / route / act / verify engine, a strong-manager model and a free classifier fast path. It is the most interesting-shaped contender yet, and on this instrument it scored 0/2 on the toy task and 0/22 across the whole real-work corpus.

Two reasons, both structural rather than a speed story:

• It cannot record a finding. Every real-work task ends by the agent writing its answer into a page variable the verifier reads. The runtime's action set — click, type, key, select, scroll, wait, focus, inspect — has no "run this JavaScript" action, so on all eleven audits it is incapable of finishing at all. The thin CLIs have an eval command, so they just do it.
• Where it can act, it does not commit the action. On TodoMVC it typed the todo text repeatedly and never pressed Enter. On a real retailer, its first click accepted a cookie banner (navigating the page), and every later attempt re-clicked the same now-stale element.

Its self-verification is also not trustworthy on its own: on 3 of 24 runs its own checker said "passed" while the independent verifier found the work was never done — on empty data. That is exactly why the pass test is independent.

It is cheap (~$0.0003/run) and its classifier fast path is real, but it is cheaper at failing. It buys no advantage over browser-relay on this workload.

The classifier fast path, measured

Underneath the decision-heavy rows sits Jev Ultrafast — a small classifier that picks the next browser action over a bounded candidate set, instead of a chat model reading the whole page each step. Its head-to-head lives in jev-tests (same TodoMVC job, one loop, one bridge, one verifier; only the decision model changes; 5 reps per arm per transport).

Jev decides about twice as fast (1.2–1.4s vs 2.5–2.6s) and on Browser Relay it is the reliable arm (5/5 against GLM's 1/5). On Playwriter the median wall time is ~40% lower. The honest boundary is offline: on 400 scored real-work decisions Jev does not beat the trivial "always inspect the page" baseline (0.68 vs 0.86 raw accuracy) — but at a confidence gate of 0.7 it covers 63% of decisions at 97% accuracy on held-out data. Use it as a gate in front of the planner, not as the sole router. (Measured via classifier.dev, since this machine has no TypeSafe key.)

Three failures I did not expect

• page-agent ships no key-press action at all, so it can never press Enter to commit a todo. It filled the field across roughly 16 attempts on 5 tabs and still reported failure. I only got it working after patching in send_keys and dispatching the key from the page's main world.
• opencli and bb-browser reach the page and claim success, but their key event carries no keyCode, so React never commits the todo. Two different projects, same upstream defect. I confirmed the root cause with a direct probe.
• Every extension-based tool kept failing to load until I realised branded Chrome silently ignores --load-extension. Chrome for Testing honours it. One line of setup unlocked five tools I had wrongly written off.

How to use it

If your job is "open a page, read something, report it", pick by speed and cost — the top rows are all fine, and they are cheap enough that the difference is noise.

If your job is a multi-step authenticated journey, do not pick from the speed table at all. Screen your real tasks against two or three candidates, run more than two reps, and watch for the false "done" — the pass test caught several of those here.

What this does not prove

TodoMVC is a controlled instrument, not a proxy for real work, and I treat the two tables as separate claims. Several rows are two-rep screening results and should not be ordered finely against each other, and the newest row is ten reps on the toy task but a single screening rep per real task. Models matter as much as harnesses, so a different model could move the middle of the table. And because I wrote both the tasks and the test, the safest thing you can do is re-run it yourself — the repo has the exact task text, the pass rules, and the raw per-run JSON.

That is the whole point: not "trust my leaderboard", but "here is a fair race you can rerun". The leaderboard lives on /solutions, and the repo is the source of truth behind it.