Solutions · Benchmarks
Web automation tool leaderboard
35 browser automation tools, one identical job, one independent pass test. Two rankings: fast-path speed and cost on a controlled microbenchmark, and real-work capability on 11 harvested tasks. They disagree, and that is the point.
The job every tool had to do
On the public TodoMVC demo app: add two to-dos (“Email supplier” and “Review invoice”), tick off only the first, switch to the Active filter, and confirm only “Review invoice” is showing with “1 item left”.
Passing is checked by independent code reading the page and the app’s saved data — never by the agent’s own say-so. The stopwatch covers only the model-plus-browser work, and failed runs are kept, never retried away.
Fast-path leaderboard
One row per harness and best-model combination, sorted by median time. Thin harnesses that let the model issue one native command per step win on speed and cost; heavyweight agent runtimes cost 10–100× more wall-clock for no accuracy gain. Rows marked scored failure reached the page but could not finish the job.
| # | Harness | Pass | Median | Tokens in/out | Cost / run |
|---|---|---|---|---|---|
| 1 | BrowserSkill Tencent/BrowserSkill · round 3 | 2/2 | 4.1s | ~6.2k / ~97 | ~$0.0005 |
| 2 | browser-relay reliefeai/browser-relay · round 3 | 2/2 | 4.3s | ~5.9k / ~95 | ~$0.0005 |
| 3 | browser-control keon/browser-control · round 3 | 1/2 | 5.5s | ~13.2k / ~180 | ~$0.0007 |
| 4 | browser-cli six-ddc/browser-cli · round 3 | 2/2 | 6.2s | ~8.6k / ~128 | ~$0.0007 |
| 5 | pinchtab pinchtab/pinchtab · round 3 | 2/2 | 8.3s | ~5.1k / ~200 | ~$0.0003 |
| 6 | browser-use browser-use/browser-use · round 1 Round 1 timing-only (tokens not recorded) | 2/2 | 8.7s | n/a | n/a |
| 7 | Browser Harness browser-use/browser-harness · round 2 | 2/2 | 9.9s | ~6.2k / ~190 | ~$0.0005 |
| 8 | agent-browser vercel-labs/agent-browser · round 3 | 2/2 | 10.1s | ~20.8k / ~120 | ~$0.0010 |
| 9 | Playwriter remorses/playwriter · round 1 Round 1 timing-only (tokens not recorded) | 2/2 | 10.1s | n/a | n/a |
| 10 | cdp-browser sids/cdp-browser · round 3 | 2/2 | 10.7s | ~6.3k / ~250 | ~$0.0003 |
| 11 | jarvis-browser bridge25/jarvis-browser · round 3 | 2/2 | 11.2s | ~8.5k / ~100 | ~$0.0005 |
| 12 | chrome-cdp-skill pasky/chrome-cdp-skill · round 6 Fastest new arrival, least reliable once promoted to 5 reps | 3/5 | 11.5s | ~21.1k / ~323 | ~$0.0083 |
| 13 | chrome-devtools-mcp ChromeDevTools/chrome-devtools-mcp · round 6 Strongest Round 6 row once reliability is counted | 5/5 | 11.8s | ~12.6k / ~89 | ~$0.0048 |
| 14 | Stagehand v4 browserbase/stagehand · round 2 | 1/4 | 13.1s | ~6.6k / 300–3,400 | ~$0.0007 |
| 15 | ego-browser citrolabs/ego-lite · round 6 | 5/5 | 14.2s | ~12.3k / ~99 | ~$0.0048 |
| 16 | webctl cosinusalpha/webctl · round 3 | 2/2 | 14.5s | ~8.2k / ~100 | ~$0.0006 |
| 17 | playwright-cli microsoft/playwright-cli · round 6 | 5/5 | 14.7s | ~21.7k / ~113 | ~$0.0083 |
| 18 | agent-chrome-cli gxbvc/agent-chrome-cli · round 3 | 2/2 | 17.4s | ~7.6k / ~180 | ~$0.0004 |
| 19 | browser-act-skills browser-act/skills · round 6 | 2/2 | 18.5s | ~6.0k / ~76 | ~$0.0009 |
| 20 | lightpanda lightpanda-io/browser · round 3 | 2/2 | 22.7s | ~7.1k / ~590 | ~$0.0006 |
| 21 | page-agent (patched: send_keys) alibaba/page-agent · round 5 Patched build; stock ships no key-press action and scores 0/2 | 2/2 | 25.3s | n/a | n/a |
| 22 | hyperagent (perform) hyperbrowserai/HyperAgent · round 6 | 3/4 | 28.3s | ~20.8k / ~3.4k | ~$0.0096 |
| 23 | opencli jackwener/OpenCLI · round 6 Scored failure: key event carries no keyCode, so React never commits the todo | 0/2 | 30.4s | ~20.8k / ~234 | ~$0.0032 |
| 24 | surf-cli nicobailon/surf-cli · round 6 | 2/2 | 31.8s | ~28.6k / ~519 | ~$0.0046 |
| 25 | browser-agent visnia-ai/browser-agent · round 4 | 2/2 | 32.5s | ~43.1k / ~1,360 | ~$0.0036 |
| 26 | bb-browser epiral/bb-browser · round 6 Scored failure: native key event carries no keyCode | 0/2 | 32.7s | ~12.6k / ~380 | ~$0.0021 |
| 27 | midscene web-infra-dev/midscene · round 6 Vision-first; belongs to the capability suite, not the speed ranking | 2/2 | 33.4s | ~444k / ~62k | ~$0.074 |
| 28 | raw-playwright baseline microsoft/playwright · round 3 | 3/4 | 42.7s | ~18.1k / ~410 | ~$0.0010 |
| 29 | adaptive-ui-runtime Rajeev-SG/adaptive-ui-runtime · round 8 Own-loop (plan/route/act/verify). Fails the job outright: no page-eval action and it re-issues the same action instead of committing it | 0/2 | 45.8s | ~3.0k / ~210 | ~$0.0003 |
| 30 | Magnitude magnitudedev/magnitude · round 2 Vision-first: most reliable on messy JS sites, slower and pricier | 4/4 | 52.6s | ~18.3k / ~2.9k | ~$0.0021 |
| 31 | Browser Use Pi browser-use/browser-use-pi · round 7 Own loop: Pi Mono agent + persistent V8 REPL + raw CDP. Three failures are false successes caught by the verifier | 7/10 | 53.5s | ~6.6k / ~0.7k | ~$0.0010 |
| 32 | BrowserCode uuuuytgg/browser-code · round 2 Own heavyweight agent loop; 10–100x more wall-clock for no accuracy gain | 2/2 | 153.0s | ~55.6k / ~3.0k | ~$0.026 |
| 33 | notte nottelabs/notte · round 3 | 2/2 | 171.8s | n/a | n/a |
| 34 | skyvern Skyvern-AI/skyvern · round 6 Autonomous multi-agent; capability suite | 2/2 | 186.5s | ~29.0k / ~4.2k | ~$0.0055 |
| 35 | browser-agent (Taylor-Bayouth) Taylor-Bayouth/browser-agent · round 5 Unrelated project that shares the visnia-ai name | 1/2 | 200.1s | ~19k–782k / ~0.5k–25k | n/a |
Real-work capability leaderboard
The same harness set scored on 11 real, provenance-backed browser tasks harvested from recorded sessions — tag inspection, script inventory, SEO audits, crawlability, canvas creation, and consent-to-add-to-cart journeys. The latency ranking and the capability ranking genuinely disagree: the fastest tool is the weakest on real work. The newest architecture — an own-loop agent that writes code against raw CDP instead of issuing one command per step — is slow on the toy task and, on this evidence, extends neither ranking.
| Harness | Fast-path | Real-work | Reps |
|---|---|---|---|
| browser-relay reliefeai/browser-relay Strongest on real work; promoted past screening | 2/2 · 4.3s | 22/33 | 1, 2, 3 |
| raw-playwright baseline microsoft/playwright Promoted past screening | 3/4 · 42.7s | 17/33 | 1, 2, 3 |
| agent-browser vercel-labs/agent-browser | 2/2 · 10.1s | 6/11 | 1 |
| cdp-browser sids/cdp-browser | 2/2 · 10.7s | 6/11 | 1 |
| Browser Use Pi browser-use/browser-use-pi Own-loop (Pi Mono + V8 REPL). Every task it passed, browser-relay also passed, so it extends neither frontier | 7/10 · 53.5s | 4/11 | 1 |
| BrowserSkill Tencent/BrowserSkill Fastest fast-path, weakest real work — the orderings invert | 2/2 · 4.1s | 2/11 | 1 |
| adaptive-ui-runtime Rajeev-SG/adaptive-ui-runtime Own-loop, non-default. Zero passes: its action vocabulary has no page-eval step, so it cannot set the finding every corpus task requires | 0/2 · 45.8s | 0/22 | 1, 2 |
Per-task difficulty
Pass counts across all harnesses. One-shot “inspect the live page and report” audits converge on almost any harness that can evaluate JavaScript; multi-step journeys and canvas construction sit above the current frontier. The anti-bot-boundary task is the hard one in the middle: three passes out of ten across all harnesses, two of them from the same tool.
| Task | Capability | Passes |
|---|---|---|
| allbirds-uk-add-to-cart-tag-check | consent → add to cart → tags | 0/10 |
| gymshark-uk-add-to-cart-tag-check | consent → add to cart → tags | 1/10 |
| puma-uk-seo-metadata-audit | SEO / structured data | 1/10 |
| tldraw-three-shape-diagram | canvas UI creation | 2/10 |
| chanel-gb-pdp-tag-inspection | tag inspection (anti-bot boundary) | 3/10 |
| rajeevg-crawlability-audit | robots.txt + sitemap | 8/10 |
| porsche-uk-script-inventory | third-party script inventory | 8/10 |
| puma-uk-script-inventory | third-party script inventory | 8/10 |
| puma-uk-tag-inspection | tag inspection | 8/10 |
| porsche-uk-tag-inspection | tag inspection | 9/10 |
| rajeevg-seo-metadata-audit | SEO / structured data | 9/10 |
Jev Ultrafast — the classifier fast path
On the same TodoMVC job with one loop, one bridge and one verifier, only the decision model changes. Over 5 runs per arm and transport (indicative, not a powered result), Jev finished every Browser Relay run while GLM managed one, and Jev decided about twice as fast per step. Numbers are produced and versioned in Rajeev-SG/jev-tests (browser-use/jev-ultrafast@452c1ad), not duplicated here. Measured via classifier.dev (jev-1.13.0), not TypeSafe — this machine has no TypeSafe key.
| Arm | Pass | Median | Decision p50 | Note |
|---|---|---|---|---|
| Jev + Browser Relay | 5/5 | 13.1s | 1218 ms | 5/5 vs GLM's 1/5 in the same 5 runs (n=5, indicative — not a powered reliability claim) |
| GLM + Browser Relay | 1/5 | 13.4s | 2453 ms | Level on time; four of five runs failed: three stopped without applying the filter, one emitted an invalid action |
| Jev + Playwriter | 4/5 | 16.1s | 1350 ms | Same 4/5 as GLM, ~40% lower median wall (n=5, indicative) |
| GLM + Playwriter | 4/5 | 27.1s | 2619 ms | Same pass count, ~11s slower median wall (n=5, indicative) |
Offline next-action replay (400 scored real-work decisions): Jev does not beat the always-inspect baseline (0.68 vs 0.86 raw), but at a 0.7 confidence gate it covers 63.3% of decisions at 96.9% held-out accuracy (95% CI 0.92-0.99). Use as a gate, not the sole router.
Read it honestly
This ranks the latency microbenchmark (one controlled instrument) and a real-work capability suite (11 harvested tasks) as two separate claims. TodoMVC never feeds the capability number, and a screening result should not be ordered finely against another screening row — the newest row is 10 reps on the toy task but only 1 rep per real task, so treat its capability cell as a screen, not a score.
The benchmark repo is the source of truth; this page is a dated snapshot of it. Full contracts, raw per-run JSON, and the exclusion log live at github.com/Rajeev-SG/web-automation-microbench.