How to test, benchmark and compare local models: reading public benchmarks honestly, and designing your own tests.
Benchmark numbers are everywhere and easy to misread. Here is what goes wrong (contamination, gaming, judge bias and staleness) and a practical checklist for telling a trustworthy score from a misleading one.
A tour of the benchmarks worth trusting (LiveBench, Aider, SWE-bench, EQ-Bench, Vals AI, Epoch, ARC-AGI and LMArena), what each measures, how it is scored, and what it is good and bad for.
Public benchmarks rarely match your actual task. Here is how to build a small, honest test of your own, borrowing the habits of the benchmarks that get it right: a private item set, objective scoring where possible, a hardened judge where not, and honest uncertainty.
A general benchmark will not tell you whether a model is good at your task. Here is what to measure, which public benchmarks are relevant, and how to build a quick personal test, for coding, RAG, reasoning, creative writing and agents.
We assume a newer model is a better one. Often the gain is marginal, sometimes it is real, sometimes the model gets worse, and sometimes there is no newer model at your size at all. Here is what actually changes between versions, in which directions, and how to decide whether to chase the latest.