AI Agent Benchmarks Explained: SWE-bench to tau-bench

An AI agent benchmark is a fixed set of multi-step tasks, run in a controlled environment, that measures whether a model plus its tools and harness can finish real work such as fixing a code repository, operating a desktop or serving a simulated customer. Unlike question-and-answer tests, agent benchmarks score the end state: did the tests pass, is the file correct, was the database updated as policy requires. Model providers now quote SWE-bench, OSWorld, Terminal-Bench and tau-bench results in launch posts and system cards, so knowing what each one measures, how it is scored and what it leaves out is essential to reading those numbers. This page is for learners, buyers and engineers who want a factual reference to the main benchmarks, built from the benchmark maintainers' own repositories and papers.

Last checked: 2026-11-15. Benchmarks are grouped by what they test and listed alphabetically within each group; order is not a ranking. This page describes benchmarks and does not report or compare model scores.

This is an independent reference page from NHI Mgmt Group based on public provider documentation; it is not sponsored or endorsed by any provider.

Key takeaways

The agent is the model plus the harness. A score belongs to a model, a scaffold (agent loop and tools), a budget and a benchmark version together; change any one and the number changes.
Versions matter. SWE-bench has Full, Lite, Verified, Multimodal and the separate SWE-bench Pro; Terminal-Bench has 1.0 and 2.0; tau-bench has moved to tau2 and later releases with fixed tasks.
Scoring is mostly execution-based. Hidden unit tests, evaluation scripts or database state checks decide success, which makes results reproducible but only as good as the tests.
Reliability needs its own metric. tau-bench reports pass^k, the chance an agent succeeds on all k repeated trials, because a single success can hide inconsistent behaviour.
Contamination is a known limit. OpenAI's SWE-bench Verified page warns that models trained on internet text are likely to be contaminated on tasks drawn from public GitHub repositories.

What is an AI agent benchmark?

A language model benchmark asks a question and checks the answer. An agent benchmark gives a model an environment (a repository in a container, a virtual machine, a set of websites, a simulated user with APIs) and a goal, lets it take many actions, and then checks the resulting state. Three parts always travel together:

Task set: the frozen list of problems, with a version or release name.
Environment and harness: the container, VM or simulator, plus the code that connects the model to tools. Terminal-Bench, for example, describes itself as a dataset of tasks plus an execution harness, and its maintainers built Harbor as the official harness for Terminal-Bench 2.0.
Grader: unit tests, scripts or state checks that decide pass or fail, and the metric reported (resolved rate, success rate, pass^k, medal rate).

For how general LLM benchmarks are read, see How to Read LLM Benchmarks. For how benchmarking differs from red teaming and evaluations, see AI Red Teaming vs Evaluations vs Benchmarks.

Which AI agent benchmarks are most cited?

Key facts for each benchmark, from the maintainer's own repository, paper or announcement. As of 15 November 2026.

Benchmark	Maintainer	What the agent must do	Size (as published)	How success is scored	Reference release or paper
Software engineering
MLE-bench	OpenAI	Complete Kaggle machine learning competitions	75 competitions; Low complexity split offered as a lite version	Any Medal (%) against the competition leaderboard, mean of at least 3 seeds	github.com/openai/mle-bench
SWE-bench (Full, Lite)	SWE-bench team	Produce a patch that resolves a real GitHub issue in a Python repository	Full test set plus a smaller Lite subset	Repository unit tests run in Docker; reported as % resolved	ICLR 2024 paper (arXiv 2310.06770)
SWE-bench Multimodal	SWE-bench team	Resolve issues in visual software domains	480 tasks in v2 (open sourced 1 September 2026)	Unit tests	ICLR 2025 paper (arXiv 2410.03859)
SWE-bench Pro	Scale AI	Resolve long-horizon software engineering tasks	V2: 642 validated tasks plus a HARD-51 subset; public and private (commercial) leaderboards	Each V2 task ships as a Harbor task directory with its own verifier and reference solution	V2 released 22 September (repository news)
SWE-bench Verified	SWE-bench team with OpenAI Preparedness	As SWE-bench	500 human-validated tasks	Unit tests; % resolved	13 August 2024
Terminal-Bench 2.0	Terminal-Bench team (hosted by Stanford, Harbor and the Laude Institute, per tbench.ai)	Complete end-to-end tasks in a containerised terminal	89 tasks in the public repository	Task tests in the container; reported as resolution rate with 95% confidence interval	1.0 launched in May; 2.0 announced with Harbor (tbench.ai)
Computer and web use
BrowseComp	OpenAI	Find hard-to-locate, entangled facts on the web	1,266 problems	Short answer matched to a single reference answer	openai.com/index/browsecomp
OSWorld (and OSWorld-Verified)	XLANG Lab	Operate real desktop and web applications in a VM	369 tasks (361 if 8 Google Drive tasks are excluded), plus 43 Windows tasks for analysis	Custom execution-based evaluation script per task	April 2024 (arXiv 2404.07972); Verified on 28 July 2025
WebArena	WebArena team (webarena.dev)	Complete tasks on self-hosted realistic websites	812 examples	Evaluation configured per example against the self-hosted sites	github.com/web-arena-x/webarena
General assistants and tool use
GAIA	Mialon, Fourrier, Swift, Wolf, LeCun and Scialom (paper authors)	Answer real-world questions needing browsing, tools and reasoning	466 questions; answers held private for 300 that power the leaderboard	Answer checked against a held answer	21 November 2023 (arXiv 2311.12983)
tau-bench (tau2 and later)	Sierra	Serve a simulated user with domain APIs while following a policy	text-v1: 50 frozen tasks per domain (airline, retail, telecom) in 6 languages, 900 calls per trial	Final database state and required outputs; pass^k for reliability	tau2 paper arXiv 2506.07982

Notes: the maintainer column names the organisation shown in the repository or paper. Terminal-Bench's home page showed Terminal-Bench 4.1 as its current leaderboard when checked on 15 November 2026; the 2.0 figures above are from its public task repository. The original tau-bench repository carries a warning that its airline and retail tasks are outdated and points to the tau2-bench repository.

What does SWE-bench Verified measure?

SWE-bench gives the agent a repository snapshot and an issue description and asks for a patch; hidden tests from the real fix decide whether it resolved the issue. OpenAI's Preparedness team and the SWE-bench authors released SWE-bench Verified on 13 August 2024 to remove tasks that could not fairly be solved.

Annotation: 93 Python developers annotated 1,699 random samples from the SWE-bench test set, three annotators per sample, keeping the most severe label.
Problems found: 38.3% of samples were flagged for underspecified issue descriptions and 61.1% for unit tests that may unfairly reject valid solutions; 68.3% of samples were filtered out in total.
Difficulty: annotators estimated fix time; the "easy" subset has 196 tasks under 15 minutes and the "hard" subset 45 tasks over one hour.
Effect: GPT-4o resolved 33.2% on Verified against 16% on the original set with the best open-source scaffold at the time, showing how much the original tasks understated ability.
Limit: the same page warns that models pre-trained on internet text are likely to be contaminated on tasks from public repositories.

Scale AI's SWE-bench Pro was built to test longer-horizon tasks; its repository states that V2 (22 September) has 642 validated tasks in Harbor format and that a commercial subset is held on a private leaderboard.

What does OSWorld measure?

OSWorld runs an agent inside a real virtual machine and asks it to complete computer tasks across web and desktop applications, file input and output, and multi-application workflows. Each task has its own execution-based evaluation script. In the original paper humans completed over 72.36% of tasks and the best model 12.24%. The public task file groups tasks by application: Chrome, GIMP, LibreOffice Calc, Impress and Writer, Thunderbird, VLC, VS Code, operating system tasks and 101 multi-application tasks. OSWorld-Verified, released on 28 July 2025, fixed issues reported by the community and added AWS support for parallel runs, and the maintainers ask that new results be compared against runs on the latest version.

What does tau-bench measure?

tau-bench emulates a conversation between a language-model-simulated user and an agent that has domain API tools and a written policy. Success means the final state is correct and the policy was followed. The current release in the tau2-bench repository runs airline, retail and telecom domains, text and full-duplex voice modes, and six languages (English, Spanish, Portuguese, Hindi, Chinese and Korean). The repository describes frozen task sets with pinned data so that runs can be compared, and post-hoc judges for voice runs that do not change the task reward.

The pass^k metric is tau-bench's main contribution to agent evaluation: pass^1 is the average success on one try, while pass^4 is the chance the agent succeeds on the same task in all four independent tries. A falling curve from pass^1 to pass^4 shows inconsistency that a single score hides.

What does Terminal-Bench measure?

Terminal-Bench tests agents on end-to-end work in containerised terminals. The 2.0 announcement describes it as a harder, better verified version, with manual and LM-assisted verification of each task, and is run through Harbor, a package that supports cloud-deployed containers and works with any agent that can be installed in a container. The public repository holds 89 task directories. Results are reported per model and agent pair, so the same model can appear several times with different harnesses.

How should you read an agent benchmark score?

Question to ask	Why it matters	Where to find the answer
Which benchmark version and subset?	Full, Lite, Verified and Pro SWE-bench scores are not comparable; OSWorld asks for comparison against the latest version	Footnotes in the model card or launch post
Which harness or scaffold?	Leaderboards such as Terminal-Bench list model and agent separately	Leaderboard entry or system card methodology
How many attempts and seeds?	MLE-bench asks for at least 3 seeds and mean plus standard error; tau-bench reports pass^k	Methodology section
What budget?	Time limits, step limits, tokens and cost change results; Terminal-Bench reports cost and tokens beside resolution rate	Leaderboard columns or footnotes
Provider-reported or independent?	Provider numbers use the provider's own setup; maintainer leaderboards use a submission process	Keep the two in separate columns, each with source and date
Was the task set public?	Public tasks risk contamination; GAIA and SWE-bench Pro keep some answers or tasks private	Benchmark paper or repository

Time horizon: a cross-benchmark measure

METR's time horizon metric, published on 19 March 2025, is the length of task (measured by how long human experts take) that an agent completes with 50% success, found by fitting a logistic curve to success against human task length. The original analysis found the 50% time horizon doubling roughly every seven months over about six years. METR states that results depend on which tasks and human baselines are used, that the projection assumes the trend continues, and that its Time Horizon 1.1 estimates announced on 29 January 2026 are now its current measurements.

What do agent benchmarks not measure?

Security of the work: a patch that passes tests can still add a vulnerability or leak a secret. Pair capability scores with the controls in the AI Coding Agents Security Guide.
Behaviour under attack: standard runs do not insert hostile instructions into issues, web pages or tool output. See the Prompt Injection Guide.
Permission discipline: benchmarks give the agent whatever access the task needs inside a sandbox; they do not score whether it asked for more than it needed. See the AI Agent Authorisation Guide.
Your environment: your codebase, tools, data and policies differ from any public task set.
Cost and latency at your scale: leaderboard cost columns reflect the benchmark run, not production traffic.

How to run your own agent evaluation

Pick tasks from your own work: 30 to 100 closed tickets, support cases or runbooks with known correct outcomes.
Write the grader first: tests, state checks or a rubric that decides pass or fail without a human reading the transcript.
Freeze a version: pin the task set, environment image and tool versions, as tau-bench's frozen releases and Terminal-Bench's datasets do.
Use an existing harness where it fits: Harbor, the tau2 framework and the SWE-bench Docker harness are open source and documented.
Run sandboxed with scoped credentials: give the agent test accounts and short-lived tokens, never production keys.
Repeat trials: at least three runs per task; report mean, spread and a pass^k style consistency figure.
Record cost, time and tokens alongside success.
Re-run on change: new model, new harness, new tools or new policy each needs a fresh run against the same frozen set.

Metrics worth tracking internally

Task success rate on the frozen internal set, per model and harness pair.
pass^3 or pass^5 consistency on the same tasks.
Cost per successful task in tokens and currency.
Policy violations per 100 tasks, such as actions outside the allowed tool list.
Human intervention rate, the share of tasks needing a person to step in.

Common failure patterns

Comparing across versions: setting a SWE-bench Verified score beside a SWE-bench Pro score as if they were the same scale.
Ignoring the harness: crediting a model for a result that depended on a custom scaffold not available to you.
Single-run results: treating one lucky run as capability when repeated trials would show a much lower pass^k.
Running evaluations with real credentials: letting an agent under test use production tokens because the sandbox was inconvenient to set up.
Stale task sets: continuing to quote an outdated release after the maintainer has fixed or replaced tasks, as the original tau-bench repository warns.

Practitioner checklist

Record benchmark name, version, subset, harness and date beside every score you quote.
Keep provider-reported and independent leaderboard results in separate columns.
Check the maintainer's repository for deprecation or "outdated tasks" notices before citing a benchmark.
Prefer metrics that show consistency (pass^k, multiple seeds) over single-run numbers.
Build a small frozen internal task set that reflects your own work.
Run all agent evaluations in sandboxes with scoped, short-lived credentials.
Track cost and human intervention next to success rate.
Pair capability results with security and authorisation testing before granting an agent production access.

AI Agent Benchmarks Explained: What SWE-bench, OSWorld and Others Measure