Benchmark (AI performance test)
Japanese ベンチマーク（AIの性能を比べる試験）
A benchmark is a shared test that gives several AI models the same set of problems and compares their scores. A number in a launch announcement only means something together with which test it was and under what conditions it was run.
A benchmark is a shared test that gives several AI models the same set of problems and turns the share they get right into a score, so the models can be compared. For example, MMLU measures knowledge with questions from 57 subjects such as mathematics, history and law, and SWE-bench measures coding skill by asking a model to fix 2,294 real issues reported on GitHub. It works like a national mock exam: because every model gets the same paper, models from different companies can be lined up side by side. When an AI company launches a new model and writes something like SWE-bench Verified 77.2%, that figure is a benchmark score.
Watch out A score depends on the test conditions. In its Claude Sonnet 4.5 announcement, Anthropic reported the same test two ways: 77.2% with the standard setup and 82.0% with a setup that uses more computation. Doing well on an exam and handling the work in your own shop are different things. When you choose a tool, treat the launch numbers as a reference only, and try a few of your own documents in it and compare the results yourself.
Source: Measuring Massive Multitask Language Understanding · SWE-bench: Can Language Models Resolve Real-World GitHub Issues? · Introducing Claude Sonnet 4.5 · Introducing Llama 3.1: Our most capable models to date · Humanity's Last Exam · ARC Prize - What is ARC-AGI? (checked 2026-10-26)