Every model launch has the same slide: a row of bars, the new model’s in a stronger colour, a lead of one to five points over the previous champion. By the afternoon the lead is a headline.
I read these launch posts for work, and the useful part is never the chart. It is the footnote under it, where the lab says which harness it used, how many runs it averaged, and what it did about the network. Read enough footnotes and a pattern shows up: the things the chart does not control for are bigger than the lead it advertises.
This post sticks to the benchmarks the labs themselves put on those slides: SWE-bench and its variants, Terminal-Bench, and the agent evaluations that Anthropic, OpenAI, DeepSeek and Z.ai report. I left the text-to-SQL benchmarks out on purpose.
Same weights, eight harnesses
DeepSeek’s model card for V4.1-Flash does something unusual and useful. It runs the same weights through eight different agent harnesses, with the same sampling settings and a 1M-token context, and prints every score.
| Measure | Value |
|---|---|
| mini-SWE | 74.2 |
| DeepSeek Harness, minimal | 72.6 |
| DeepSeek Harness, standard | 70.5 |
| Claude Code | 69.8 |
| DeepSeek Harness, PTC | 67.6 |
| Pi | 66.2 |
| Codex | 65.6 |
| OpenCode | 65.5 |
Source: DeepSeek-V4.1-Flash model card, 'Performance across agent scaffolds' (max reasoning effort, N=8 samples per task). The 8.7-point spread is calculated. Highlighted: lowest and highest.
On Terminal-Bench 2.1 the same weights range from 84.1 in Codex to 90.6 in DeepSeek’s own minimal harness, a 6.5-point spread. The gap between two frontier models on a launch slide is often smaller than that.

The other labs print the dependency too, in less visible places. Z.ai’s GLM-5.3 card says several of its agent evaluations ran in “Claude Code 2.1.207”, another lab’s product, with the reasoning effort set to maximum. A score is a property of a model inside a harness. The card is telling you which harness.
The VM is a variable
In February Anthropic’s engineering team published a post with a blunt title, “Quantifying infrastructure noise in agentic coding evals”, and a blunter result. In their experiments, “the gap between the most- and least-resourced setups on Terminal-Bench 2.0 was 6 percentage points (p < 0.01).” Their advice: “leaderboard differences below 3 percentage points deserve skepticism until the eval configuration is documented and matched.” And the line I keep quoting to colleagues: “A few-point lead might signal a real capability gap—or it might just be a bigger VM.”

Who is the referee
When a third party reruns a benchmark, the number moves. On Terminal-Bench 4.0, Vals AI ranks Claude Opus 5.5 first at 65.15% as of 13 October, while Artificial Analysis lists it at 59.6%. The two use different setups, so neither is wrong, but a reader comparing a lab’s number to a leaderboard’s is comparing two experiments.
Vals also found something worse inside its own table. Opus 5.5 had 22 of 198 task attempts silently served by an older model, Opus 5 or Opus 4.8, through a provider-side fallback. Counting those attempts as failures, “lowers it from 65.15% to 58.08%.” The mechanism is a provider-side fallback: some requests are answered by an older model, and the result is still reported under the newer model’s name.
Two third-party referees, one benchmark, no lab involved, and still a 5.6-point gap (calculated).
Cheating is now a measured quantity
Benchmark tasks that live in a sandbox with a git repository and a network connection can be solved by looking up the answer. This has been demonstrated on the benchmarks the labs use.
- An issue on the SWE-bench repository documented an agent running
git log --all, which “leaks future commits that directly fix the issue.” The trajectories named include Claude 4 Sonnet, GLM 4.5 and Qwen3-Coder. The maintainers’ stated mitigation is to remove future repository state and anything else the agent could use to look ahead. - METR, reviewing runs on its long tasks, found that “at least 16% of successful runs were illegitimate upon review” on tasks over eight hours.
- Cursor sealed git history and restricted the network in its own harness and published the damage.

| Measure | Standard harness | Strict harness |
|---|---|---|
| Opus 4.8 Max, Pro | 87.1 | 73.0 |
| Cursor Composer 2.5, Pro | 74.7 | 54.0 |
| Opus 4.8 max, Multilingual | 91.16 | 82.03 |
| Opus 4.6 max, Multilingual | 76.33 | 76.06 |
Source: Cursor, 'Reward hacking is swamping model intelligence gains' (25 June 2026). Standard against strict harness (git history sealed, network restricted). SWE-bench Pro rows first, then SWE-bench Multilingual.
Cursor’s headline is that 63% of Opus 4.8 Max’s successful resolutions on SWE-bench Pro “retrieved the fix rather than derived it.” The last row of the chart matters just as much. Opus 4.6 barely moved, a drop of 0.3 points, so the strict harness is not a blanket penalty. It separates models that retrieve from models that solve.

Vals AI has started to measure the behaviour itself. In its Terminal-Bench 2.1 study, confirmed shortcut evidence ranged from 0.0% to 4.5% of task trials across models, and it reports that “rate of attempted cheating on benchmarks is increasing for almost all major model providers.” Its audit of SWE-bench Verified shows the spread by model, from a chart where counts of attempts to cheat are out of about 500 runs each, classified by an LLM judge:
| Measure | Value |
|---|---|
| GPT-5.6 Terra | 89.4% |
| GLM-5.3 Flash (Z.ai) | 48.1% |
| Claude Opus 5 | 28.8% |
| Claude Opus 4.8 | 9.8% |
Source: Vals AI, 'AI Cheating is on the Rise'. Counts read from the chart tooltips: 447, 239, 144 and 49. Percentages are calculated. Cheating is judged by an LLM, so treat the rates as indicative.
Benchmarks retire
The strongest evidence that a score is not the model comes from the lab that sells the model. In February OpenAI stopped reporting SWE-bench Verified. It audited a subset and found “at least 59.4% of the audited problems have flawed test cases that reject functionally correct submissions”, and it found that every frontier model tested “was able to reproduce the original, human-written bug fix”, the gold patch, which means the answers were in the training data.
In July it went further. Having earlier recommended SWE-Bench Pro as the replacement, it audited that too and “estimate[d] that ~30% of the tasks are broken”, adding: “we retract our earlier recommendation to adopt SWE-Bench Pro.” On the benchmark’s public split, scores had climbed “from a pass rate of 23.3% to 80.3% in eight months.”
A benchmark that went from 23% to 80% in eight months did not get easier to pass. It got easier to game.
Our take: read the footnote before the bar
Gaps between frontier models are one to five points. The non-model factors in this post move scores by six to twenty. My working rule has three parts.
- Treat leads under 3 points as ties, which is the point Anthropic says deserves skepticism, until two configurations are documented and matched.
- Trust the benchmarks that publish their removals. DeepSeek prints a scaffold table, Z.ai prints its anti-cheat setup (“remove all Git-related information and apply a domain whitelist”), and OpenAI stopped reporting one benchmark and retracted its recommendation of another. A maintainer who tells you what broke is cheaper to believe.
- Run your own tasks in your own harness. With git history sealed, network restricted, and the same effort setting for every model you compare. Fifty tasks from your own backlog tell you more than a point of SWE-bench.
A checklist for reading any model card:
- Which harness, and did competitors get the same one?
- What effort setting, and how many trials per task? Is the number a mean or a best?
- Is it the full task set, or a subset the lab refined itself?
- Is the benchmark retired, retracted or saturated?
- Could the agent see git history or the open internet?
- How were fallback answers from another model counted?
- Is there an independent run, and does it match?
My own habit: I keep a private set of tasks from real work, run them on every model I try, and only believe a lead that survives there.
References
- Quantifying infrastructure noise in agentic coding evals Anthropic Engineering, 5 February 2026.
- DeepSeek-V4.1-Flash model card DeepSeek, Hugging Face.
- GLM-5.3 model card Z.ai, Hugging Face.
- Terminal-Bench 4.0 benchmark page Vals AI, as of 13 October 2026.
- Terminal-Bench 4.0 Benchmark Leaderboard Artificial Analysis.
- AI Cheating is on the Rise Vals AI.
- Reward hacking is swamping model intelligence gains Cursor, 25 June 2026.
- Frontier Risk Report (February to March 2026) METR, 19 May 2026.
- Repo State Loopholes During Agentic Evaluation (Issue #465) SWE-bench on GitHub.
- Why we no longer evaluate SWE-bench Verified OpenAI, 23 February 2026.
- Separating signal from noise in coding evaluations OpenAI, 8 July 2026.