How the good benchmarks actually work

The previous guide covered how benchmark numbers mislead. This one looks at the benchmarks that get the design right, so you can recognise a trustworthy source and borrow its ideas. None of these is perfect, and each says so; that candour is part of why they are worth citing.

Objective capability: LiveBench

LiveBench (paper) measures general capability across six areas (maths, reasoning, coding, language, data analysis, instruction following). Its two good decisions: questions are drawn from recently published material (fresh maths competitions, recent arXiv papers and news) and new items are withheld before release, which limits contamination; and scoring is entirely objective against ground-truth answers, with no model acting as judge. It versions releases by date. Its own repository notes a weakness worth knowing locally: the local-inference path is less maintained than the API one, and a model that refuses on safety grounds is scored wrong, which can understate it.

Coding, two different questions: Aider and SWE-bench

Aider’s polyglot leaderboard asks whether a model can write and edit code unaided across six languages. It uses the 225 hardest Exercism exercises (the ones few reference models could solve, to avoid a ceiling), scores them with the exercises’ own unit tests, and gives the model a second try after showing it the failing test output (so it also measures self-correction). It separately reports how often the model produced a correctly formatted edit, a direct instruction-following signal, and reports cost per solved problem. Its limitation: Exercism problems are public, so assume some contamination.

SWE-bench asks the harder, more realistic question: can an agent resolve a real GitHub issue in a real repository so that the project’s tests pass. Scoring is execution-based and strict: the fix must make the previously failing tests pass and keep the passing ones passing (no regressions), run in a per-issue container for reproducibility. The catch is contamination and scope. The popular “Verified” subset is public and, by 2026, heavily contaminated, and Epoch’s analysis notes a good agent scaffold can move the score by up to twenty points, so the number reflects the harness as much as the model. The newer SWE-bench Pro answers this with held-out and private commercial repositories that cannot be trained on.

Subjective quality, done carefully: EQ-Bench

Creative writing and emotional intelligence have no unit test, so EQ-Bench (paper) uses an LLM judge, and hardens it. Prompts are engineered to expose weaknesses rather than flatter; comparisons run in both orders and are averaged to cancel position bias; outputs are length-standardised so verbosity cannot leak in; model identities are hidden. It reports run-to-run variance and recommends several iterations. Two ideas are worth stealing: its “Slop Score” is a cheap, objective count of over-used LLM clichés against a public list, and “Judgemark” is a test of whether a model is any good as a judge at all. It states plainly that judged results are “roughly indicative but not absolute truth” and are not comparable across different judge models.

Independent re-runs: Vals AI

Vals AI is a third-party evaluator for economically useful domains (law, finance, tax, healthcare, coding). Its value is independence and a three-tier data split: a public validation set to show the shape of the tasks, and a permanently private test set used for published results, which never leaves their hands so it cannot leak into training. It mixes objective checks with rubric judging, runs stochastic benchmarks several times, and reports the standard error, while stating honestly that its error bars do not capture variation from prompts, seeds, or deployment settings.

Auditable reasoning: Epoch AI and FrontierMath

Epoch AI is the most auditable operator. It runs on the open Inspect framework, publishes its exact task definitions, runs each model many times (for example sixteen times on some tests) and shows confidence intervals. Its FrontierMath (paper) uses original, unpublished problems from dozens of research mathematicians, with answers that are exact and “guessproof” (complex enough that random guessing succeeds under one percent of the time), plus a held-out subset. It is candid that a re-review found about one problem in twenty needed fixing, a useful reminder that even expert benchmarks carry an error rate.

Novelty, not knowledge: ARC-AGI

ARC-AGI (ARC-AGI-2 paper) tests fluid reasoning: solving novel abstract grid puzzles from a few examples, designed to be easy for people and hard for machines. Every task is new, scored by exact match, with a private test set that is never published (so it cannot be trained on) and a verified human baseline (tasks are confirmed solvable by people). It treats cost per task as a first-class axis, comparing a few dollars for a human against far more for a model, which is the right instinct for local AI, where the real budget is memory, tokens per second and electricity.

Human preference: LMArena

LMArena (paper) is the canonical human-preference benchmark: people compare two anonymous answers and vote, and a Bradley-Terry model turns millions of votes into ratings with bootstrap confidence intervals. It measures something real that objective tests miss, but, as the previous guide covered, raw preference rewards length and formatting and the leaderboard has proven gameable through selective disclosure. Read its style-controlled numbers, and never treat preference as proof of correctness.

The pattern

Across all of these, the trustworthy designs share a few habits: a contamination defence (private or freshly sourced items), objective scoring where the task allows it and a hardened, validated judge where it does not, honest uncertainty, versioning, and a published list of their own limitations. Those are exactly the habits to copy when you build a test of your own.