Why benchmarks are hard, and how to read them

Every model launch comes with a table of benchmark scores, and almost all of them are read wrongly. A number on its own (“scores 90 on X”) tells you very little without knowing who produced it, how, and whether the test still measures what it claims to. This guide is about reading those numbers well. The companion guides cover how the good public benchmarks actually work and how to design your own tests.

Four things go wrong, again and again.

1. Contamination: the test leaked into the training data

A benchmark measures reasoning only if the model has not already seen the answers. The moment a test is published to the web, it starts turning up in the next generation’s training data, and the score stops measuring the skill and starts measuring memory.

This is not hypothetical. SWE-bench Verified, long the reference for coding agents, is built from public GitHub issues, and by 2026 models demonstrably reproduce the “gold” fix verbatim; an analysis by Epoch AI found every frontier model shows signs of contamination on it, which is a large part of why OpenAI stopped reporting it for launches. The benchmarks that stay trustworthy are the ones built to resist this: LiveBench draws questions from material published after training cut-offs and withholds new items before release; FrontierMath uses original, unpublished problems from invited mathematicians; ARC-AGI and Vals AI keep a permanently private test set that is never published. The lesson for reading a score: a high number on a years-old public benchmark is close to meaningless for a current model.

2. Gaming: optimising the metric instead of the skill

Any measure that becomes a target stops being a good measure (Goodhart’s law). With benchmarks this takes two forms.

The subtle form is stylistic. The human-preference leaderboard LMArena collects millions of pairwise votes, and when its makers added “style control” to separate substance from presentation, the rankings moved materially: response length and markdown formatting (headers, bold, lists) were rewarding models as much as the quality of the answer. A model can climb by being longer and more decorated, not better.

The blunt form is strategic. A 2025 study, The Leaderboard Illusion, documented that the same preference leaderboard could be gamed through selective disclosure (testing many private model variants and publishing only the best one’s score) and unequal data access, producing large gains on the leaderboard’s own distribution without a matching gain in general quality. The statistics were sound; the process around them was not. The lesson: a score reflects the incentives of whoever produced it, so ask what they could gain.

3. What is being measured, and how it is scored

“Scores 85” means nothing until you know what kind of measurement produced it. There are three broad kinds, and they are not comparable.

• Objective scoring checks the answer against verifiable ground truth: a unit test passes, a number matches, a grid is exactly right. LiveBench scores entirely this way, explicitly without an LLM judge; Aider’s leaderboard runs the code against its own unit tests; SWE-bench requires the fix to make failing tests pass without breaking passing ones. This is the most robust kind.
• Judged scoring asks another model (or a person) to rate quality against a rubric. It is unavoidable for genuinely subjective work like creative writing, but it carries real biases: models tend to prefer their own style (self-preference), a judge and a candidate from the same family share blind spots (correlated error), and judges reward length and formatting just as human voters do. EQ-Bench, which judges creative and emotional tasks, is candid that its results are “roughly indicative but not absolute truth”, runs each comparison in both orders to cancel position bias, and even publishes a separate test (Judgemark) for whether a model is any good as a judge in the first place.
• Human preference aggregates which answer people liked, as LMArena does with a Bradley-Terry model over pairwise votes. It captures something real that objective tests miss, but it measures preference, not correctness, and should never be cited as evidence a model is more accurate.

4. Staleness: benchmarks rot

A benchmark is a snapshot. Models saturate it (everyone scores 95, so it no longer separates them), it gets contaminated, or the task stops mattering. Two long-standing references, the Hugging Face Open LLM Leaderboard and Papers With Code, were retired in 2025; SWE-bench Verified lost credibility in early 2026. The good benchmarks treat their scores as perishable: LiveBench versions its releases by date, and Epoch AI runs each model many times and pins results for auditability. A score with no date attached is a score you cannot trust.

Worth being honest about: even carefully built benchmarks carry errors. FrontierMath’s own authors, re-reviewing a subset, found roughly one problem in twenty needed correction. Expert-written does not mean flawless.

How to read a benchmark number

Before you act on any score, ask:

1. Who ran it? An independent evaluator (LiveBench, Vals AI, Epoch) or the organisation that released the model? Treat a vendor’s own figure as a claim until someone independent reproduces it.
2. When, and which version? A figure with no date, and no benchmark version, is not usable. Recency matters as much for the benchmark as for the model.
3. What exactly was measured? Which benchmark, which variant (SWE-bench Verified and SWE-bench Pro are very different), and under what conditions (thinking mode on or off, tools allowed, how many attempts)?
4. How was it scored? Objective, judged, or human preference. Do not compare across those kinds.
5. Is it contamination-resistant? Public test set and an old model generation means assume leakage.
6. Is uncertainty reported? A single run with no confidence interval is weak evidence. Models are stochastic; good evaluators run several times and report a range.
7. At what cost? A win that needs ten times the memory or compute is not a win for someone running locally.

This is the standard we hold ourselves to. Our model rankings and the recommendations dataset cite the source and date of every figure, label vendor claims as claims, and prefer independent, transparent evaluators (the “Sources we trust” section on the recommendations page lists them). A benchmark is a starting point for your own testing, never a verdict, which is exactly why the next step is learning to run a test of your own.