GPQA, AIME, SWE-bench, Elo: What LLM Benchmarks Actually Measure
August 9, 2026 · AI API Index
People cite model rankings as if "best" were a single number. It isn't. Every benchmark measures a narrow slice of ability, and a model that tops one table can be thoroughly mid on another. This is a tour of the benchmarks behind our rankings page, with my opinion on how much weight each deserves.
The aggregate indices
Artificial Analysis Intelligence Index
Artificial Analysis re-runs a battery of evaluations (knowledge, math, coding, instruction following) against each model's public API under consistent settings, then combines the results into one score. Since it's measured independently rather than reported by vendors, it's the default ranking metric on this site. The current leader is Claude Opus 5 at 63.1. One caution from watching this index for a while: gaps under about 2 points are noise. Don't switch providers over 1.5 points.
LMArena Elo
Crowdsourced blind voting. Users get two anonymous answers to the same prompt and pick the one they like better, which produces a chess-style rating. What it really measures is how answers feel: helpful, well-formatted, confident. That's genuinely useful if you're building consumer chat. It's also gameable by confident tone, and it tells you little about hard reasoning.
Knowledge and reasoning tests
GPQA Diamond
About 200 graduate-level science questions written so that a smart person with a search engine still can't answer them. PhDs outside their own field manage around 34%. Frontier models are now above 90%, which means the test has saturated. It still separates mid-tier models from frontier ones. Between two frontier models, it no longer tells you much.
AIME
Competition math, 15 problems, integer answers, easy to grade. Top models sit at 98 to 100%. Effectively solved, and I'd ignore small differences here entirely.
Humanity's Last Exam
Around 2,500 expert-written questions across dozens of fields, built specifically because the older tests were saturating. Frontier scores currently range from about 40 to 58%, so there's still headroom, and right now it's one of the few knowledge tests that meaningfully separates the top models.
MMLU-Pro and MMMU
MMLU-Pro is a harder ten-choice successor to the classic MMLU. MMMU covers multimodal reasoning, meaning questions that require actually reading charts, diagrams and photos. If vision input matters to your product, MMMU is the column to look at; otherwise skip it.
Coding
SWE-bench Verified
My favorite of the bunch, because it resembles real work. The model receives an actual GitHub repository plus a real filed issue, and has to produce a patch that passes the project's test suite. That requires navigating a codebase, editing files and running tools, so it captures agentic ability that trivia-style benchmarks miss. One big caveat: scores depend heavily on the agent harness, and vendors report numbers from their own best setups. Cross-vendor comparisons are softer than they look. Self-reported figures carry an asterisk on our tables.
LiveCodeBench
Competitive-programming problems collected continuously after each model's training cutoff, precisely so the answers can't be in the training data. Good complement to SWE-bench: this one isolates raw algorithmic problem-solving, SWE-bench tests working inside a messy repo.
Caveats that apply to all of them
- Contamination. Public benchmarks leak into training sets. Treat scores on older static tests as upper bounds. Continuously refreshed tests are more trustworthy.
- Self-reporting. A vendor-reported score used the vendor's preferred settings. Marked with * here; independent re-measurements are used wherever they exist.
- Run settings. The same model scores differently depending on reasoning budget, tools and temperature. Seeing a 2-point disagreement between two sources is normal, not a scandal.
- Saturation. Once everyone scores 95%, the test has stopped measuring anything. GPQA and AIME are there. HLE and SWE-bench aren't yet.
How I'd actually use all this: rank by the independent aggregate, sanity-check with whichever benchmark is closest to your workload, then decide based on price versus performance and your own eval rather than the leaderboard.