An LLM benchmark is a fixed set of tasks with a scoring rule, used to measure and compare what large language models can do, and a benchmark score only means something when you know the task set, the scoring method and the conditions the model was run under. Model announcements, model cards and leaderboards are full of percentages and ratings, and the same model can show different numbers in different places. This page explains the main benchmark types, how scores are produced, the difference between provider-reported and independent results, the known pitfalls such as contamination and saturation, and a checklist for reading a benchmark table. It is for anyone comparing models, from learners to buyers and engineers. It describes benchmarks and does not rank any model.
Last checked: 2026-11-15.
This is an independent reference page from NHI Mgmt Group based on public provider documentation; it is not sponsored or endorsed by any provider.
Key takeaways
- A score is a result of a setup, not a fixed property. Prompting, number of attempts, tools, reasoning budget and the test harness all change the number.
- Separate provider-reported from independent results. Providers report their own runs in model cards; independent groups such as Artificial Analysis, Epoch AI and LMArena run or collect results under their own rules.
- Know what kind of benchmark it is. Knowledge tests, coding tasks, agentic tasks and human preference votes measure different things and are not interchangeable.
- Contamination and saturation limit older tests. Public test items can end up in training data, and once top models cluster near the ceiling a benchmark stops separating them.
- Your own evaluation decides fit. Public benchmarks narrow a shortlist; a test set built from your own tasks shows whether a model works for you.
What is an LLM benchmark?
A benchmark has three parts: a dataset of tasks (questions, coding issues, multi-step jobs), a protocol for running the model on them (prompt format, tools allowed, number of attempts), and a metric (accuracy, percent resolved, a rating). Change any of the three and the score changes. That is why a benchmark name alone, such as "SWE-bench", does not tell you enough: you also need the variant, the harness and the settings.
Benchmarks answer a narrow question well: how did this model do on these tasks under these conditions? They do not directly answer whether a model is right for your product, your data or your risk tolerance. The live NHI Mgmt Group guide AI Red Teaming vs Evaluations vs Benchmarks explains how benchmarks differ from internal evaluations and red teaming; this page goes deeper on reading the numbers themselves.
What types of benchmark are there?
The examples below are well-known benchmarks with their published design. They illustrate the categories; they are not a complete list.
| Type | Example | What it contains | Metric |
|---|---|---|---|
| Broad knowledge | MMLU (Hendrycks et al., 2020) | 57 tasks across subjects such as elementary mathematics, US history, computer science and law | Accuracy |
| Expert reasoning | GPQA (2023) | 448 multiple-choice questions in biology, physics and chemistry, written to be hard to answer with web search | Accuracy; the paper reports 65% for domain experts (74% discounting clear mistakes) and 34% for skilled non-experts with web access |
| Frontier knowledge | Humanity's Last Exam (Center for AI Safety and Scale AI) | 2,500 closed-ended, verifiable questions across more than a hundred subjects, finalised on 3 April 2025 | Accuracy, plus calibration error from the model's stated confidence |
| Software engineering | SWE-bench and SWE-bench Verified | Real GitHub issues from Python repositories: 2,294 instances in the original set, 500 in the human-filtered Verified subset | Percent resolved (the model's patch passes the repository tests) |
| Agentic tasks | Terminal and tool-use suites | Multi-step tasks in a sandbox where the model must plan and call tools | Task success rate |
| Human preference | LMArena (formerly Chatbot Arena) | Anonymous side-by-side comparisons voted on by users | A rating derived from pairwise votes |
| Composite index | Artificial Analysis Intelligence Index; Epoch Capabilities Index | Several benchmarks combined into one number with stated weights or a statistical model | Index score |
Agent benchmarks have their own design questions (environments, tool access, cost per task). See AI Agent Benchmarks Explained.
How is a benchmark score produced?
The same model and benchmark can produce different numbers because of choices made when running it. These are the settings to look for in footnotes.
| Setting | What it means | Why it moves the score |
|---|---|---|
| Shots | Zero-shot (instructions only) or few-shot (worked examples in the prompt) | Examples can lift scores on format-sensitive tasks |
| Attempts (pass@1, pass@k) | pass@1 counts the first answer; pass@k counts a task as solved if any of k attempts succeeds | pass@k is higher than pass@1 for the same model |
| Majority vote or best-of-n | Several samples, then a vote or a selector picks one | Uses more compute than a single sample |
| Reasoning budget | How many thinking tokens a reasoning model may use | Higher budgets often score higher at higher cost; see Reasoning Models Explained |
| Tools | Whether search, code execution or other tools were allowed | Tool use changes what the task is testing |
| Harness or scaffold | The code that wraps the model, feeds tasks and parses answers | Different scaffolds on the same model can give different results |
| Temperature and max output | Sampling randomness and output length cap | Truncated answers fail; randomness changes single runs |
| Repeats | How many times the evaluation was run and averaged | Single runs carry more noise |
| Subset and version | Which slice of the benchmark (for example Verified or Lite) and which release | Subsets differ in size and difficulty |
The harness effect is documented. OpenAI's SWE-bench Verified announcement (13 August 2024) reported that GPT-4o resolved 33.2% of Verified samples with the Agentless scaffold, compared with 16% on the original SWE-bench, showing how much both the task set and the scaffold matter. The SWE-bench site now offers a Bash Only view in which all models run in the same mini-SWE-agent environment to reduce scaffold differences.
Provider-reported and independent results
Benchmark numbers reach readers from two kinds of source. Keep them in separate columns when comparing, and record the date of each.
| Source type | Who runs it | What to check |
|---|---|---|
| Provider-reported | The model developer, published in model cards, system cards and launch posts | Footnotes on settings, harness, attempts and reasoning budget; whether competitor numbers were run by the provider or copied from elsewhere |
| Independent evaluator | A third party that runs the benchmarks itself under one method for all models | The published methodology, version of the index, and the date the model was tested |
| Crowd preference | A platform aggregating user votes | Category (text, coding, vision), vote counts and the statistical method |
Three independent sources are widely cited. Each publishes its own method; summaries below are from their pages as of 15 November 2026, listed alphabetically.
- Artificial Analysis: its Intelligence Index (version v5.0.1 at the time of checking) combines nine evaluations in four equally weighted categories: knowledge work, coding, science and general. It describes running the evaluations with its own harnesses, zero-shot prompting, temperature 0 for non-reasoning models and 0.6 for reasoning models unless the lab recommends otherwise, and pass@1 scoring aggregated across repeats. It states the index is primarily text-based and English-language.
- Epoch AI: its benchmarking hub lists results for 99 benchmarks and 428 tracked models, mixing benchmarks it administers internally with external data. Its Epoch Capabilities Index combines many benchmarks into one scale using a Bayesian model with uncertainty estimates, and the hub flags when a result rests on limited pre-launch testing.
- LMArena: users compare two anonymous models' responses and vote; identities are revealed after the vote. Leaderboards are split by category, including text, WebDev, vision, document and search. The 2024 Chatbot Arena paper reported over 240,000 votes at that time and found that crowd votes agreed well with expert raters.
Independent and provider numbers for the same benchmark often differ because of the settings above. Neither is automatically wrong; they answer slightly different questions. The model's own card usually explains its setup: see Model Cards and System Cards Explained.
What are the common pitfalls?
- Contamination: if test items appear in training data, scores reflect memory rather than ability. A NAACL 2024 study (arXiv 2311.09783) found that ChatGPT and GPT-4 reproduced the exact hidden answer option on MMLU in 52% and 57% of cases, which the authors read as a sign of contamination. Newer benchmarks use private test sets, fresh items or post-cutoff data to reduce this.
- Flawed items: when OpenAI and 93 developers reviewed 1,699 sampled SWE-bench tasks, 68.3% were filtered out, mainly for underspecified issues or unit tests that could reject valid solutions. The result was the 500-task Verified set. Older benchmarks may contain wrong or ambiguous answers that cap the achievable score.
- Saturation: once leading models score near the top, small differences are within noise. Harder successors such as GPQA and Humanity's Last Exam were built for this reason.
- Cherry-picking: a launch table shows the benchmarks a developer chose. Check which common benchmarks are absent and whether settings match across the models compared.
- Mismatched settings: comparing a pass@1 score with a pass@k or majority-vote score, or a tool-assisted run with a no-tools run, is not a like-for-like comparison.
- Ignoring cost and time: a higher score at a much higher reasoning budget can cost more per task. See Model Pricing Explained.
- Treating ratings as accuracy: a preference rating measures which answers voters liked, which can reflect style and length as well as correctness.
Practitioner checklist
- Identify the benchmark and variant: name, subset (for example Verified, Lite, Diamond) and version.
- Find the source and date: provider card, independent evaluator or crowd platform, and when the result was produced.
- Read the footnotes: shots, attempts, reasoning budget, tools, harness, repeats.
- Check that compared models used the same setup: if not, treat the comparison as indicative only.
- Look for uncertainty: error bars, confidence intervals or repeat counts. Small gaps without them may not be meaningful.
- Ask what the benchmark measures: knowledge, coding, agentic work or preference, and whether that matches your use.
- Consider contamination and saturation: how old the benchmark is and whether top scores cluster at the ceiling.
- Weigh cost and latency: score per unit of cost or time can matter more than the headline score.
How to choose which benchmarks matter for you
Choose by factual criteria tied to your use case:
| If your use is | Benchmarks that relate to it | What to add yourself |
|---|---|---|
| Coding assistant or coding agent | SWE-bench Verified and other repository-level coding suites | Tasks from your own repositories and languages |
| Research or analysis | GPQA, Humanity's Last Exam, long-document tasks | Questions from your domain with known answers |
| Agents that use tools | Agentic and terminal suites | Your tools, permissions and failure handling |
| Customer-facing chat | Preference leaderboards by category | Your tone, policy and refusal requirements |
| Long documents | Long-context retrieval and reasoning tests | Your documents at real length; see Context Windows Explained |
Then build a small private evaluation set: 50 to 200 representative tasks with expected outputs, scored the same way for every candidate model, and rerun it when a model version changes. Track accuracy on that set, cost per task, latency, refusal rate and error rate. For production monitoring after selection, see the LLM Observability Guide.
Do benchmarks measure safety?
Capability benchmarks do not measure whether a model is safe to deploy in your setting. Providers publish safety and misuse evaluations in system cards, and independent groups run their own, but these are separate from the capability scores discussed here. Security testing of an application built on a model needs its own work: see the LLM Security Guide and LLM Jailbreaks Guide.
Standards and references
- Artificial Analysis: Intelligence benchmarking methodology (accessed 2026-11-15)
- Epoch AI: Benchmarking hub (accessed 2026-11-15)
- LMArena: How it works (accessed 2026-11-15)
- Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference (accessed 2026-11-15)
- SWE-bench Team: SWE-bench (accessed 2026-11-15)
- OpenAI: Introducing SWE-bench Verified (accessed 2026-11-15)
- Hendrycks et al.: Measuring Massive Multitask Language Understanding (accessed 2026-11-15)
- GPQA: A Graduate-Level Google-Proof Q&A Benchmark (accessed 2026-11-15)
- Center for AI Safety and Scale AI: Humanity's Last Exam (accessed 2026-11-15)
- Investigating Data Contamination in Modern Benchmarks for Large Language Models (accessed 2026-11-15)
Related NHI Mgmt Group resources: AI Red Teaming vs Evaluations vs Benchmarks, AI Agent Benchmarks Explained, Model Cards and System Cards Explained, Context Windows Explained.