Designing your own local-LLM tests

The most useful eval is almost never a public leaderboard. It is a small test built around the work you actually do, on material a model has never seen. This guide shows how to build one that tells you the truth, using the design habits of the benchmarks in the previous guide. You do not need a research budget; you need a tight question and some discipline.

1. Define one capability, and its pass condition, first

Before writing a single test item, write down the one thing you are measuring and the exact condition for a pass. “Is it good at coding” is not testable; “resolves this class of bug in my repository so these tests go green” is. The strong benchmarks all narrow hard: Aider selects only its hardest problems to avoid everyone scoring full marks, and EQ-Bench picks items for their power to separate models rather than for coverage. A tight definition is what makes the result mean something.

2. Build a private item set (this is the decision that matters most)

Every benchmark trusted in 2026 defends against contamination, because a test the model has seen measures memory, not skill. You have an advantage the public benchmarks do not: your own material. Use your repositories, your documents, your support tickets, your prompts, things that were never on the public web.

Three rules:

• Keep the answer key out of any file the model or an agent can read.
• Never paste your test items into a model you are also ranking (that is how you contaminate your own benchmark).
• If you must use public items, assume contamination and treat the score as a best case, not a true one.

This mirrors what the serious benchmarks do: private test sets at Vals AI and ARC-AGI, unpublished problems at FrontierMath, private commercial repositories at SWE-bench Pro.

3. Prefer scoring that does not need an opinion

The most robust tests avoid a judge entirely by engineering tasks that have a checkable answer: a unit test that passes, an exact number or string, schema-valid JSON, a retrieval hit on a known document. LiveBench scores against ground truth with no judge; Aider and SWE-bench run real unit tests; FrontierMath uses exact answers complex enough that guessing fails.

Borrow SWE-bench’s two-sided check for anything with code: a test that must pass after the change to prove the fix works, and a set that must stay passing to prove nothing regressed. And add a “guessproof” check: if a blank or generic answer scores a point, your item is broken.

If that checkable answer means running code the model wrote, treat the code as hostile. Model output can loop forever, exhaust memory, read your files, or reach the network, so execute it in an isolated sandbox: an embedded interpreter or a container with no access to the host, module loading and imports denied, and hard limits on wall-clock time, memory and recursion depth. A small, dependency-free engine with those limits is enough, and it is the difference between a test harness and a remote-code-execution hole. Test the sandbox itself with deliberately hostile inputs (an infinite loop, deep recursion, a file read, a network call) and confirm each is contained before you trust a single score.

4. If the task is genuinely subjective, harden the judge, and validate it

Some work (prose, tone, helpfulness) has no objective answer, so you need a rubric or an LLM judge. Judges are usable but biased, and you must design against the biases:

• Self-preference: a model favours its own style, so never let a model judge a contest it is in. Use a different, ideally stronger, model as the judge.
• Correlated error: a judge and a candidate from the same family share blind spots, so vary the judge’s family.
• Style and length (Goodhart): judges reward verbosity and markdown, as LMArena’s style-control work showed, so standardise length and strip formatting before judging.
• Position bias: judge each pair in both orders (A then B, and B then A) and average, as EQ-Bench does.

Most important, and most often skipped: validate the judge before you trust it. This is what EQ-Bench’s “Judgemark” is for. Give the judge responses whose relative quality you already know and check that it actually ranks them correctly. A judge you have not validated is a number generator.

5. For subjective quality, prefer pairwise verdicts over absolute scores

Asking a judge (or yourself) for an absolute “7 out of 10” is unstable; people and models anchor differently from day to day. Asking “is A better than B” is far more reliable. Collect pairwise verdicts and fit a Bradley-Terry model (the method behind LMArena ratings, and nothing more than maximum likelihood over win rates) to turn them into a ranking with confidence intervals. Anchor the scale to a couple of fixed reference models so numbers stay comparable as you add contenders.

This is also the honest home for your own judgement. If you are the one who knows what “good” means for your work, your own pairwise verdicts over your own material are a legitimate, uncontaminated signal, with no model judging quality at all. Report it for what it is (one person’s preference on their own tasks), keep it separate from objective scores, and let the confidence intervals speak: on a handful of comparisons they will be wide, and “too close to call” is a true answer.

6. Report uncertainty, and say what it misses

Models are stochastic; a single run is an anecdote. The good evaluators run each model several times and report a range: Epoch runs many iterations with confidence intervals, Vals reports a standard error over repeated runs. Do the same: run each model multiple times (mandatory once temperature is above zero), report the mean with an interval, and state how many runs.

Then copy Vals AI’s honesty and say plainly what your error bars do not capture: the choice of prompts, the random seed, the harness, and, for local models, the quantisation level. A Q4 and an FP16 run of “the same model” are different subjects and must be labelled as such.

7. Match the design to the task

• Coding: a real repository and its own tests; require both the fix-passes and nothing-regresses checks; test more than Python; give one retry after showing the failing output; score edit-format compliance separately. Fix and document your agent scaffold, since a good one can swing the score by twenty points.
• RAG and grounding: questions whose answers live in a known private corpus; score both the retrieval hit and faithfulness (is the cited source actually right); include “the answer is not in the corpus” items to catch confident hallucination.
• Creative: prompts designed to expose weaknesses rather than flatter; pairwise with a validated external judge; and a cheap objective “slop” check as a sanity signal alongside the judged scores. The slop check needs no judge: have the model write a short sample (say 200 words), count hits against a curated list of over-used LLM phrases and filler, and normalise per 1,000 words, where lower is better. It measures only cliché density, not quality, so it cannot be gamed by a judge’s taste, it runs in milliseconds, and the phrase list is easy to keep current.
• Agentic and tool use: long, multi-step tasks with an objective end-state check (did the world reach the goal?); a standardised tool environment and a cap on turns so comparisons are fair; log cost, latency and turn count as first-class results.

8. Make cost a first-class metric

A model that wins only at ten times the memory or compute has not won for a local user. ARC-AGI treats cost per task as the fairest way to compare systems, and Aider reports cost per solved problem. For local models this is concrete: tokens per second, VRAM used, and the quantisation it took to fit. Report quality and cost together.

9. Automate it, and assume it will rot

Version your item set, pin the harness and the judge model and version, record the exact prompts, decoding settings and quantisation level, and script the whole run so a result is reproducible. Version the test itself so a score always names which version produced it.

Finally, assume your benchmark is gameable and perishable, because all of them are: The Leaderboard Illusion showed even a statistically sound test gets gamed, SWE-bench Verified aged into contamination, and FrontierMath’s own authors found errors in their problems. Build in a correction path, refresh the items periodically, and publish the test’s known weaknesses. On a question as consequential as “which model should I run”, stating what your test cannot tell you is part of telling the truth.

---

This is the same standard behind our model recommendations and rankings: sourced, dated, honest about uncertainty, and never dressed up as a verdict.