A benchmark is a fixed set of tasks with a scoring rule, used to measure an AI model and compare it against others. Every number you see in a model launch — “80.9% on SWE-bench Verified”, “#1 on Arena” — is a benchmark result. They exist because “which model is smarter?” is unanswerable without agreeing on the questions first.
The main kinds
Knowledge and reasoning exams (GPQA, MMLU, AIME) test facts and problem-solving with verifiable answers. Task benchmarks score models doing real work — SWE-bench has models fix actual GitHub issues; Terminal-Bench has agents complete jobs in a shell; OSWorld tests computer use from screenshots. Preference leaderboards (LMArena’s Elo system) skip fixed answers entirely: humans vote blind between two model outputs, and models are ranked like chess players. Media models — image and video generators — rely on this style almost exclusively, since there’s no ground truth for “the better-looking clip”.
Why scores mislead
Three failure modes recur. Contamination: benchmark questions leak into training data, so a model “knows” answers it never reasoned out. Saturation: once frontier models cluster above 90%, remaining differences are noise — which is why benchmarks keep getting replaced by harder successors (MMLU → GPQA, SWE-bench → SWE-bench Pro). Harness sensitivity: the same model scores differently depending on scaffolding, tools and retry budgets, which is how two labs can both truthfully claim the lead on “the same” benchmark. A model can also be fine-tuned to a benchmark’s format without gaining general ability — teaching to the test, at industrial scale.
How to actually read them
Treat benchmarks as a screening filter, not a verdict. Check the version and date (SWE-bench Pro is not SWE-bench Verified), prefer numbers from a neutral harness over vendor-reported ones, and weight the benchmark that resembles your workload — a coding lead says little about writing quality. Our best-of rankings cite the specific benchmark behind every claim for exactly this reason. The final test is always the same: run the model on your own tasks. A benchmark tells you which models are worth that hour, nothing more.