Solutions · AI & Agent Systems
Agent Benchmark Matrix
Compare current agent models where credible public evidence overlaps, then inspect where coverage is still missing. No ranking across unlike runs, and no invented composite score.
A blank always means no ingested public result, never a poor score. Looking for the quality-versus-cost view instead? See the LLM Pareto Frontier.
How this page is built
The default comparison deliberately holds back benchmark and model combinations with too little public evidence to compare fairly. Those combinations are not deleted: they appear in full under Coverage and in the “tracked, awaiting current results” list, and they graduate into the comparison automatically once enough results are ingested.
Snapshot generated 13 Sept 2026 · 23 benchmarks · 22 models · 111 reported results · 3 names queued for triage.
Benchmarks tracked
19 P0 · 23 total
Models in registry
22
Result records
111 reported
Evidence split
12 official · 99 vendor
Benchmarks with at least 3 tracked models and models with at least 2 results in that set. Ultra-sparse combinations are held back here and shown in full under Coverage; nothing is inferred to fill a gap.
Comparable evidence
8 models × 7 benchmarks · 38/56 cells populated (68%)
A dash means no ingested public result for that combination — not a failed model. Select a benchmark heading to open its leaderboard below.
| Model | |||||||
|---|---|---|---|---|---|---|---|
Benchmark leaderboard
Terminal-native tasks an agent must complete end-to-end from a shell.
Ranked leaderboard10 comparable runs
version 2.1 · base set · Accuracy
- Terminal-Bench 2.1 Accuracy, ranked only within runs sharing version 2.1 · base set · Accuracy.
- 90.6Accuracy
- 88.8Accuracy
- 88.3Accuracy
- 88.2Accuracy
- 87.9Accuracy
- 87.4Accuracy
- 86.6Accuracy
- 85.6Accuracy
- 84.7Accuracy
- 84.3Accuracy
| Rank | Model / system | Accuracy | Harness | Evidence | Source |
|---|---|---|---|---|---|
| 1 | 90.6 | Terminus 2 | Vendor | Source | |
| 2 | 88.8 | Codex CLI | Vendor | Source | |
| 3 | 88.3 | Kimi Code | Vendor | Source | |
| 4 | 88.2 | Unspecified | Vendor | Source | |
| 5 | 87.9 | Terminus 2 | Vendor | Source | |
| 6 | 87.4 | Codex CLI | Vendor | Source | |
| 7 | 86.6 | Unspecified | Vendor | Source | |
| 8 | 85.6 | Codex CLI | Vendor | Source | |
| 9 | 84.7 | Codex CLI | Vendor | Source | |
| 10 | 84.3 | Claude Code | Vendor | Source |
Separate protocol group 22 comparable runs
version 2.1 · subset “Claude Code harness” · Accuracy
- Terminal-Bench 2.1 Accuracy, ranked only within runs sharing version 2.1 · subset “Claude Code harness” · Accuracy.
- 80.4Accuracy
- 74.6Accuracy
Separate protocol group 32 comparable runs
version 2.1 · subset “Codex CLI harness” · Accuracy
- Terminal-Bench 2.1 Accuracy, ranked only within runs sharing version 2.1 · subset “Codex CLI harness” · Accuracy.
- 79.6Accuracy
- 78.9Accuracy
Tracked, awaiting current results
Important benchmarks where the public record does not yet cover enough of the tracked models to compare. They are tracked and clickable, not dropped — and they graduate into the comparison above automatically once enough results are ingested.
Coverage
Coverage answers “which of my tracked models have actually been evaluated broadly?”, and where the public record is thin. It is deliberately not a quality score.
0 benchmark-official · 16 vendor-reported
0 benchmark-official · 5 vendor-reported
0 benchmark-official · 5 vendor-reported
0 benchmark-official · 5 vendor-reported
0 benchmark-official · 4 vendor-reported
0 benchmark-official · 4 vendor-reported
0 benchmark-official · 13 vendor-reported
2 benchmark-official · 2 vendor-reported
| Model | Benchmarks with a result | Official | Vendor-reported |
|---|---|---|---|
| GPT-5.6 Sol | 14 / 19 | 0 | 16 |
| GPT-5.6 Terra | 5 / 19 | 0 | 5 |
| GPT-5.6 Luna | 5 / 19 | 0 | 5 |
| GLM-5.3 | 5 / 19 | 0 | 5 |
| GLM-5.3 Flash | 4 / 19 | 0 | 4 |
| DeepSeek V4.1 Flash | 4 / 19 | 0 | 4 |
| Kimi K3 | 13 / 19 | 0 | 13 |
| Qwen3.8 Max | 3 / 19 | 2 | 2 |
Workload axes covered
- Cross-application work12
- Tool / API / MCP8
- Browser / web6
- Enterprise SaaS6
- Deep research / browsing5
- Policy / multi-turn3
- Desktop / CUA2
- Skills / harness use1
- Terminal / coding1
Known audit caveats
- OSWorld 2.0 — 18 major, 25 minor findings (v2026.08.08). Audit
Advanced controls
Maintenance and source diagnostics
Source freshness
- OSWorld 2.0 project page · manual · checked 2026-09-13 — Vendor/benchmark tables read manually; no machine-readable result file published yet.
- OpenAI GPT-5.6 release benchmarks · manual · checked 2026-09-13 — Vendor-official computer-use, tool-use and research tables with footnoted harness settings.
- Z.ai GLM-5.3-Flash model card · manual · checked 2026-09-13 — Vendor-official benchmark table; harness annotations vary per benchmark.
- DeepSeek-V4.1-Flash model card · manual · checked 2026-09-13 — Vendor-official table including a DeepSeek-harness comparison row.
- Kimi K3 model card · manual · checked 2026-09-13 — Vendor-official table with explicit per-benchmark harness footnotes (Kimi Code / Claude Code / Codex).
- Terminal-Bench leaderboard · manual · checked 2026-09-13 — Leaderboard submissions pin dataset, agent, model and reasoning effort.
- WebArena-Verified repo · pending · checked 2026-09-13 — Result artefacts published per release; automated adapter not yet enabled.
- τ³/τ² submissions tree · pending · checked 2026-09-13 — Structured submission.json per run; adapter planned.
- TheAgentCompany experiments · pending · checked 2026-09-13 — Versioned per-run task-level JSON; adapter planned.
- AppWorld raw leaderboard JSON · pending · checked 2026-09-13 — Canonical machine-readable source; adapter planned.
- Artificial Analysis evaluations · manual · checked 2026-09-13 — Independent evaluator; treated as a separate protocol wherever it reimplements a benchmark.
- Parsewave OSWorld 2.0 audit · manual · checked 2026-09-13 — Independent audit of release v2026.08.08: 18 major + 25 minor findings across 43 tasks.
Unresolved benchmark names (2)
- GAIA-2 — Referenced but no canonical benchmark identity established yet; queue for explicit triage.
- "τ-bench" (unversioned) — Ambiguous generation (τ vs τ² vs τ³); requires explicit version before it can be ingested.
Unresolved model names (1)
- GPT-5.6 Sol Ultra — Same model evaluated under a higher reasoning/agentic config; stored as a config on gpt-5.6-sol rather than a separate model.
Candidate benchmark registry
39 benchmarks recorded. 6 p0 · 17 p1 · 11 p2 · 3 rejected · 2 superseded.
- Mind2Web (original) — superseded: Static offline tasks; largely replaced by Online-Mind2Web and WebArena-Verified.
- MiniWoB++ — superseded: Toy-scale; retained only as a historical sanity check.
- CoWorkBench (vendor-internal) — rejected: Vendor-internal benchmark with no public tasks/protocol; cannot be a canonical comparable source.
- JobBench (vendor-internal) — rejected: Vendor-internal; may be stored as vendor-reported evidence but not a public benchmark.
- Vision2Web (vendor-internal) — rejected: Vendor-internal; no public tasks or evaluation protocol.
Missing-cell meanings
- no_public_result — No public result found for this model.
- tracked_not_ingested — Benchmark tracked, but no current-model result ingested yet.
- incompatible_protocol — This model has results for this benchmark under a different version, subset or protocol, so it cannot be ranked in this column.
- filtered_out — No comparable result under the current filters.
Provenance key
Official benchmark-official ·Paper paper ·Independent independent ·Vendor vendor-reported
How to read this
A benchmark score is never just “model → number”. It is a combination of benchmark, benchmark version, subset, metric, model revision, harness, tool and observation mode, and reasoning or step budget. Two runs of the same model under different harnesses stay separate records, and are only ranked together inside a strictly comparable group.
Missing cells stay missing. A blank is not a zero and is never filled from a neighbouring model, a family member, a vendor claim or a different benchmark. Some blanks mean “not publicly reported”; others mean “not ingested yet”. Both are stated rather than hidden.
There is deliberately no cross-benchmark “agent intelligence” score. Percentages from different benchmarks use different tasks, denominators and protocols and are not additive. Per-benchmark ranks, percentiles and coverage counts are shown instead. Provenance is labelled on every cell: benchmark-official data is the strongest class, a vendor-reported score is marked as such, and independent reimplementations are kept as their own protocols.