Evaluating for your use case

A model that tops a general leaderboard can still be the wrong choice for your work, because the leaderboard was not measuring your work. The way to choose well is to evaluate for the specific thing you will actually do. This guide goes use case by use case: what matters, which public benchmarks are worth reading, the quick personal test to run, and the pitfall to avoid. It assumes you have read how to read benchmarks and how to design a test. Our recommendations are organised by these same tasks.

General chat and knowledge

What matters: broad knowledge, instruction following, and giving answers you would actually keep.

Public benchmarks: LiveBench is the soundest objective general measure (fresh items, no judge). MMLU-Pro and GPQA Diamond are widely reported knowledge-and-reasoning scores, useful as a proxy but vendor-reported for most models. LMArena captures human preference, which matters for “do people like its answers”, but read the style-controlled numbers and never as proof of correctness.

Your quick test: keep a running file of the real questions you ask week to week, with a note on what a good answer looks like. Run a candidate model over them and read the answers yourself. Twenty of your own questions tell you more than any leaderboard.

Pitfall: general scores hide task weakness. A model can top a knowledge board and still be a poor coder or a careless summariser, so do not let one headline number stand in for the specific thing you need.

Coding

What matters: this splits in two, and the split is the single most common mistake. Code generation (write a function from a description) is a different skill from agentic software engineering (navigate a real repository, change several files, keep the tests green). A model can be strong at one and weak at the other.

Public benchmarks: Aider’s polyglot leaderboard tests unaided writing and editing across six languages and reports edit-format compliance, close to real local-coding use. SWE-bench tests agentic issue resolution; prefer the newer SWE-bench Pro or an independent re-run over the contaminated plain Verified. LiveCodeBench is a contest-style generation measure. Hold generation and agentic scores apart: a high HumanEval and a modest SWE-bench belong to the same model routinely.

Your quick test: take a real repository you know, with its own test suite. Give the model a genuine issue and require two things, as SWE-bench does: the failing test must pass after its change, and the previously passing tests must stay passing. Test in the languages you actually use, not only Python. Fix and record your agent scaffold, since a good one can swing the result by up to twenty points, so otherwise you are benchmarking the harness.

Pitfall: trusting a generation score for agentic work, or a single-language score for a polyglot codebase.

Reasoning and maths

What matters: multi-step problem solving where the final answer is either right or wrong.

Public benchmarks: AIME and MATH-500 are objective and widely reported; GPQA Diamond covers graduate-level science reasoning; FrontierMath is the frontier (research-level, guessproof) but near-floor for most models; ARC-AGI tests novel reasoning rather than recalled knowledge and is a good check for genuine generalisation.

Your quick test: reasoning is the easiest use case to test honestly, because the answers are checkable. Collect problems from your domain with a single correct answer, keep the answer key out of the model’s reach, and score exact matches. Add a guessproof check: if a generic or blank answer scores, the item is broken.

Pitfall: maths and reasoning benchmarks are often reported in “thinking” mode with large token budgets, so a headline score may not reflect how you would run the model day to day. Note the mode.

RAG and grounding

What matters: answering over your documents, faithfully. Two things are being tested at once: did it retrieve the right source, and did it answer truthfully from that source without inventing.

Public benchmarks: long-context retrieval measures like RULER tell you whether a model can find a fact buried in a long context, which is a proxy for grounding, not a faithfulness measure. There is no widely trusted public faithfulness benchmark at the sizes most people run locally, which is exactly why a personal test matters here.

Your quick test: build questions whose answers live in a known private corpus. Score two things separately: the retrieval hit (did it surface the right document) and faithfulness (is the cited source actually correct, not invented). Crucially, include “the answer is not in the corpus” items, to catch a model that confidently makes something up rather than saying it does not know. Keep the corpus private so it cannot have been trained on.

Pitfall: judging the model alone. In a RAG system the retriever and the embedding model do half the work; a weak retriever makes a strong model look bad, and vice versa. Test the system, and change one part at a time.

Creative writing

What matters: quality that no unit test can capture: voice, structure, taste. This is irreducibly subjective, so evaluate it accordingly, and be honest that the result is a preference, not a measurement.

Public benchmarks: EQ-Bench judges creative and emotional tasks with a hardened LLM judge and is candid that its results are “roughly indicative, not absolute truth”. Its “Slop Score”, a deterministic count of over-used LLM clichés, is a cheap objective sanity signal with no judge involved.

Your quick test: the most honest approach is your own pairwise verdicts. Give two models the same prompt (ones designed to expose weakness, not flatter), read both blind, and record which you prefer. Fit those pairwise verdicts into a ranking over time. This is your taste on your material, with no model judging quality, which is the right home for a subjective call. Run the cheap cliché check alongside as a guardrail.

Pitfall: treating a judged creative score as a verdict, or letting response length and formatting stand in for quality (both human voters and LLM judges reward verbosity). Standardise length before you compare.

Agentic and tool use

What matters: completing a multi-step task by using tools, recovering from failure, and reaching a correct end state, over a long horizon.

Public benchmarks: the Berkeley Function-Calling Leaderboard measures tool-calling accuracy; tau-bench and Terminal-Bench measure multi-step agent tasks; SWE-bench is the reference for agentic software engineering. Across all of them, remember that the scaffold (the harness around the model) heavily shapes the score.

Your quick test: define a real multi-step task with an objective end-state check, did the world actually reach the goal, not did the transcript look plausible. Standardise the tool environment and cap the number of turns so comparisons are fair, and log cost, latency and turn count as first-class results, since an agent that succeeds slowly or expensively may not be worth it.

Pitfall: scoring the transcript instead of the outcome, and forgetting that tool use results reflect the harness as much as the model. Fix the harness across models, or you are not comparing like with like.

A note that runs through all of them

Whatever the use case, the habits from the previous guide hold: test on material the model has not seen, prefer an objective check where the task allows one, report how many runs and the spread, and label the quantisation you tested, because a Q4 and an FP16 run of the same model are different subjects. The point of evaluating for your use case is not a tidy number; it is confidence that the model will do your work well.