AI Model Scoring Benchmark
GitVelocity scores every pull request with a large language model. This page covers which model scores your PRs today, what happens to your scores when that model changes, and how we benchmark models to decide. The Current scoring model section and the Latest benchmark results and Changelog at the bottom are updated with every change. The rest of the page describes how we work, and holds from one model to the next.
Current scoring model
Updated October 15, 2026.
- Default: Claude Sonnet 5.5, since October 2, 2026. If your organization has never picked a model in Settings → AI Scoring, Sonnet 5.5 scores your new PRs.
- Reference: Claude Fable 5. Every model is benchmarked against it. It is not offered for scoring.
- Latest benchmark: the October 2026 edition (32 PRs, 3 runs per model). See Latest benchmark results.
How the selectable models compare on the latest benchmark:
- Sonnet 5.5 (the default) agrees with the reference more closely than Sonnet 4.6, the previous default, costs slightly less per call and takes about half the time. It leans a little low, about 6% below the reference on average.
- Opus 5.5 is the closest match to the reference, and the only model measurably closer than the default. It has no measurable lean and costs about 2.1× the default per call. Pick it if you want the closest agreement with the reference.
- Sonnet 4.6 scores about 16% above the reference. Pick it if keeping your score history on one scale matters more than the accuracy gain.
- Haiku 5.5 costs about 5% of the default per call on the same Anthropic key. It is about as close to the reference as the default, but noisier, and it scores large features higher.
- On your own OpenAI key: GPT-6 Sol costs about half the default per call and is about as close to the reference. GPT-6 Luna is one of the cheapest models measured, with noisier individual scores. GPT-5.6 Terra and GPT-5.6 Luna stay selectable but agree less closely and lean high.
- On your own OpenRouter key: GLM 5.3 Flash costs about 3% of the default per call and is about as close to the reference, but its individual scores are noisier, mostly on small PRs. GLM 5.1 and Kimi K3 are also lower-cost options. Both agree less closely than the default and lean high.
Opus 5, Opus 4.8, Opus 4.7, Opus 4.6 and Sonnet 5 are also selectable. Claude Platform on AWS can route Opus 5, Opus 4.8, Opus 4.7, Sonnet 5.5, Sonnet 5 and Sonnet 4.6 through your own AWS account.
What changes for you
When the model that scores your PRs changes, new PRs are scored by the new model and scores already recorded are not re-scored. That happens when GitVelocity moves its default and your organization uses the default, or when someone in your organization picks a different model. If you picked a specific model, changes to the default do not move you.
Different models do not score on one scale, so your trend lines can step up or down at the date of the change. The trend charts mark that date with a dashed line topped by a diamond, and a step there comes from the model, not your team. The step is usually a percentage of the score rather than a fixed number of points, so it is larger in points on larger PRs. The measured size of every change to the default is in the Changelog.
To put your whole history on one scale, see Do I have to rescore my old PRs when GitVelocity switches to a new AI model?
How we handle model changes
Every model scores on its own scale. Two models that rank PRs in nearly the same order can still differ by 10–20% in the level they score at. A new model will move your numbers, and we will keep adopting new models, for two reasons:
- Older models get retired. Model providers retire models on a schedule, so staying on one model forever is not an option. Pinning a model postpones a change of scale; it does not avoid one.
- Newer models usually agree with the reference more closely. Holding scores steady by staying on an older model would mean keeping a known error.
What we can control is the direction of each change, how large it is allowed to be, and whether you hear about it. This is the bar a new default has to clear:
- It moves toward the reference. A new default must agree with the reference at least as closely as the model it replaces, by mean absolute deviation on the same corpus. We will not change the default to a model that lands further from the reference, however much cheaper or faster it is.
- Its average lean stays within ±10% of the reference.
- It ranks PRs the way the reference does (correlation 0.98 or higher). A model that reorders PRs is a different kind of measurement, not a recalibration, and we would not make it the default.
- We measure the step before you see it. For every default change we publish the size of the step on the same PRs, add a dated entry to the Changelog, and mark the date on your trend charts.
How big a step to expect. If the old and new defaults each sit within ±10% of the reference, the step between them can be as large as about 20%, and steps between two models well inside that band are smaller. For scale, the same model scoring the same PR twice typically moves it by about 4–8%, so a step of a few percent is hard to see in a trend line, and a step above 10% is not.
These are the rules we decide by, not a promise that your numbers will never move. If you need one scale across a period, pin a model in Settings → AI Scoring or rescore that period on one model.
How we benchmark
To choose the default and decide which other models to offer, we score a fixed set of real pull requests with every candidate and compare each one against a fixed reference model on four dimensions:
- Accuracy — how far each model's score lands from the reference model's score for the same PR
- Stability — how much a model's score for the same PR moves between runs
- Cost — uncached list-price cost per scoring call
- Speed — observed API latency per scoring call
The reference model
The reference is the strongest model we can run on the corpus. It is used only to measure other models against and is not offered for scoring, because it cannot be measured against itself and it costs several times the default per call. The reference changes rarely, and only with a new benchmark edition; each change is recorded in the Changelog.
A note on "accuracy." The benchmark has no human-labeled ground truth. "Accuracy" on this page means agreement with the reference model, not proof that a score is objectively correct. Read every accuracy figure as agreement with the reference.
How a run is set up
- Corpus. A fixed set of pull requests from private production repositories, chosen for diversity across size, language and change type. Your own PRs are never used.
- The production prompt. The benchmark builds its prompt through the same code production uses, including the pre-computed effective-line and file counts that set the Effort Scale Factor.
- The production request. Each model gets the request production sends it: the same temperature, thinking and reasoning-effort settings.
- One rubric. Every model in an edition is scored on the same rubric version.
- Repeated runs. Every model scores every PR three times, so stability can be measured and each PR's score is the mean of its runs.
What the numbers mean
- Mean Absolute Deviation (MAD) — averaged over the corpus, how many points a model's score lands from the reference's score for the same PR. Lower is better. It is an average, so individual PRs land closer or further; the max deviation column shows the worst PR.
- Bias — the average signed difference. A bias of +3 means the model came out about three points high across the corpus as a whole, not that it added three points to every PR.
- Correlation (r) — whether the model ranks PRs the way the reference does. High correlation with a large bias means the ordering is right on a shifted scale.
- Stability (CV) — for each PR, the sample standard deviation of a model's scores divided by their mean. We report the average over the corpus, the worst PR, and how many PRs varied by more than 10%. GitVelocity scores each PR once, so this is how repeatable the number you see is. CV is large when a score is near zero, where a one-point move is a large percentage.
- Latency — median wall-clock time for one scoring call from a single client. It is an observation, not an SLA.
When we say one model is "closer" than another. MAD is an average over a few dozen PRs, so a small gap between two models can be noise. We say one model agrees with the reference more closely than another only when a paired bootstrap over the corpus puts the whole 95% interval of the difference on one side of zero. Otherwise we say "about as close".
How we decide
Before each run we write down the bar a candidate has to clear, so it cannot be fitted to the result. For a new default, that bar compares it with the current default on the same run: agreement with the reference, bias, stability, cost per call, latency, and every call returning a usable score with no refusals or truncations. A candidate for a selectable, non-default option is compared with the closest comparable model already offered.
When a candidate fails a gate and we adopt it anyway, we say which gate it failed and why we overrode it, in the Changelog.
What the benchmark does not tell you
- Whether the reference is right. Every accuracy figure is agreement with the reference model, which has not been validated against human reviewers.
- Your production costs. Costs are uncached list price. Production caches the shared prompt prefix, so real per-call costs are lower, and provider rates change.
- Fine-grained differences. With a few dozen PRs, gaps of a few tenths of a point in MAD are usually within noise. Where we claim one model is closer than another, a paired bootstrap backs it. Figures for a single size tier rest on a handful of PRs and are directional only.
- Latency guarantees. Latency is one client on one day, not an SLA.
- Your scores. Benchmark figures come from the fixed corpus, never from your own PRs.
Switching models
Open Settings → AI Scoring and pick a model in the Model Selection card. Models from Anthropic run on the same Anthropic key as the default. Models from OpenAI or OpenRouter need your own key for that provider, connected first:
1. Connect your key in the API Keys card: the OpenAI row for the GPT options, the OpenRouter row for the OpenRouter-backed ones.
Click Edit on the row for your provider — OpenAI or OpenRouter — to paste that provider's key. Only the last four characters are ever shown back to you.
2. Switch the scoring model in the Model Selection card. Every bring-your-own option stays disabled until its provider's key is saved.
PR reviews route through your own key as soon as you pick a bring-your-own model. Your list may show more options than this screenshot.
Switching changes how future PRs are scored, not PRs already scored, so comparing scores from before and after a switch is comparing two scales. See What changes for you. GitVelocity is free because you bring your own key, so scoring costs are paid to your provider directly.
Latest benchmark results
October 2026 edition, run on October 1, 2026, against the Fable 5 reference. Every model scored the same 32 pull requests, 3 runs each, on rubric v7 with the production prompt and request. Haiku 5.5 was scored on October 7, its release day, and eight low-cost OpenRouter models on October 15, all with the same PRs, prompt and benchmark code and measured against the same October 1 Fable 5 runs. GLM 5.3 Flash and GLM 5.3 run with minimal reasoning, because both refuse requests with reasoning off. Each configuration has all 96 calls scored.
| Corpus | Mix |
|---|---|
| Size | 6 tiny (<50 lines), 8 small (50-150), 9 medium (150-500), 6 large, 3 XL |
| Language | TypeScript (17), Rust (8), Ruby/Rails (7) |
| Change type | 17 features, 8 fixes, 4 config changes, 3 refactors |
| Runs | 3 per model per PR: 96 scored calls per configuration, 2,688 across 28 configurations |
Fable 5 averages 22.9 points on this corpus, so a bias of +1 is about 4% of the average score.
Pricing
Every rate checked October 1, 2026 (Haiku 5.5 on October 7, the low-cost OpenRouter models on October 15).
| Model | Provider | Input $/1M | Output $/1M | Available to you |
|---|---|---|---|---|
| Fable 5 | Anthropic | $10.00 | $50.00 | No — used as the benchmark reference |
| Opus 5.5 | Anthropic | $4.00 | $20.00 | Yes |
| Opus 5 | Anthropic | $5.00 | $25.00 | Yes |
| Opus 4.8 | Anthropic | $5.00 | $25.00 | Yes |
| Opus 4.7 | Anthropic | $5.00 | $25.00 | Yes |
| Opus 4.6 | Anthropic | $5.00 | $25.00 | Yes |
| Sonnet 5.5 | Anthropic | $2.00 | $10.00 | Yes — the default |
| Sonnet 5 | Anthropic | $2.00 | $10.00 | Yes |
| Sonnet 4.6 | Anthropic | $3.00 | $15.00 | Yes — the previous default |
| Haiku 5.5 | Anthropic | $0.10 | $0.50 | Yes |
| GPT-6 Sol | OpenAI (direct) | $2.00 | $10.00 | Yes — needs your own OpenAI key |
| GPT-6 Luna | OpenAI (direct) | $0.10 | $0.50 | Yes — needs your own OpenAI key |
| GPT-5.6 Terra | OpenAI (direct) | $2.00 | $12.00 | Yes — needs your own OpenAI key |
| GPT-5.6 Luna | OpenAI (direct) | $0.20 | $1.20 | Yes — needs your own OpenAI key |
| GPT-6 Astra | OpenAI (direct) | $10.00 | $50.00 | No — evaluated and declined |
| GPT-5.6 Sol | OpenAI (direct) | $4.00 | $20.00 | No — evaluated and declined |
| GLM 5.3 Flash | OpenRouter | $0.15 | $0.50 | Yes — needs your own OpenRouter key |
| Kimi K3 | OpenRouter | $0.664 | $10.00 | Yes — needs your own OpenRouter key |
| GLM 5.1 | OpenRouter | $0.965 | $3.032 | Yes — needs your own OpenRouter key |
| Kimi K2.6 | OpenRouter | $0.434 | $1.828 | No — retired |
| MiniMax M3 | OpenRouter | $0.30 | $1.20 | No — evaluated and declined |
| MiMo v2.6 Pro | OpenRouter | $0.435 | $0.87 | No — evaluated and declined |
| GLM 5.3 | OpenRouter | $1.40 | $4.40 | No — evaluated and declined |
| MiMo v2.6 Flash | OpenRouter | $0.14 | $0.28 | No — evaluated and declined |
| LongCat 2.0 | OpenRouter | $0.30 | $1.20 | No — evaluated and declined |
| Hunyuan 3 | OpenRouter | $0.0825 | $0.33 | No — evaluated and declined |
| DeepSeek V4.1 Flash | OpenRouter | $0.0156 | $0.72 | No — evaluated and declined |
Haiku 5.5's rates are for prompts up to 100,000 tokens ($0.50 / $2.50 above that). The largest prompt in the corpus is about 45,000 tokens, so every call here is billed at the lower rate. Claude Platform on AWS routes are the same models on a different route, so they were not benchmarked separately. Opus 5.5 and Haiku 5.5 are available through a direct Anthropic key only.
OpenRouter rates are aggregated across upstream providers and move between reads. GLM 5.3 is listed at the rate of Z.ai's own endpoint, because cheaper hosts may not serve every request. Kimi K2.6's rate had risen to $0.65 / $3.41 by October 15, and its figures below use the October 1 rate. Official pricing sources:
- Anthropic model pricing
- OpenAI API pricing — standard short-context tier
- Kimi K3 on OpenRouter
- GLM 5.1 on OpenRouter
- GLM 5.3 Flash on OpenRouter
Reference agreement
32 PRs per model, each the mean of 3 runs (96 scored calls per row), measured against Fable 5's mean for the same PR. Sonnet 4.6 appears twice: (production) is how GitVelocity calls it, at temperature: 0 with thinking off, and (thinking on) is how earlier editions benchmarked it.
| Model | MAD vs Fable 5 | Max deviation | Bias | Correlation (r) |
|---|---|---|---|---|
| Opus 5.5 | 0.95 | 3.67 | −0.03 | 0.998 |
| GLM 5.3 Flash | 1.14 | 3.33 | −0.63 | 0.998 |
| Opus 5 | 1.15 | 3.33 | +0.96 | 0.999 |
| GPT-6 Luna | 1.43 | 3.87 | −0.79 | 0.996 |
| Opus 4.7 | 1.57 | 8.67 | +1.15 | 0.996 |
| Sonnet 5.5 (default) | 1.59 | 5.47 | −1.38 | 0.996 |
| Opus 4.8 | 1.64 | 5.67 | +1.33 | 0.997 |
| Opus 4.6 | 1.85 | 7.67 | +0.79 | 0.992 |
| MiniMax M3 | 1.86 | 9.67 | +0.80 | 0.990 |
| GPT-6 Sol | 1.99 | 7.67 | +0.84 | 0.992 |
| Haiku 5.5 | 1.99 | 8.00 | +1.69 | 0.996 |
| MiMo v2.6 Pro | 2.28 | 10.33 | +1.54 | 0.995 |
| Kimi K3 | 2.91 | 15.00 | +2.60 | 0.992 |
| GPT-5.6 Terra | 2.93 | 11.33 | +2.72 | 0.994 |
| GLM 5.1 | 3.22 | 12.67 | +2.95 | 0.992 |
| GLM 5.3 | 3.27 | 12.33 | +3.18 | 0.995 |
| GPT-5.6 Sol | 3.28 | 13.00 | +3.22 | 0.996 |
| Sonnet 4.6 (thinking on) | 3.37 | 9.67 | +3.37 | 0.996 |
| Sonnet 5 | 3.45 | 13.00 | +3.41 | 0.995 |
| GPT-6 Astra | 3.49 | 14.67 | −3.42 | 0.988 |
| MiMo v2.6 Flash | 3.51 | 17.33 | +2.79 | 0.980 |
| Sonnet 4.6 (production) | 3.73 | 11.67 | +3.69 | 0.992 |
| GPT-5.6 Luna | 3.98 | 17.33 | +3.97 | 0.991 |
| Kimi K2.6 | 4.07 | 14.33 | +3.82 | 0.987 |
| LongCat 2.0 | 4.43 | 12.67 | +3.91 | 0.989 |
| Hunyuan 3 | 4.48 | 16.33 | +4.06 | 0.992 |
| DeepSeek V4.1 Flash | 4.64 | 16.00 | +4.64 | 0.986 |
Every model ranks the 32 PRs much as Fable 5 does (r between 0.980 and 0.999), so the differences are mostly about scale, not ordering. Against the default, by paired bootstrap:
- Opus 5.5 agrees with Fable 5 more closely than Sonnet 5.5: MAD 0.64 lower, interval −1.16 to −0.14, closer on 19 of 32 PRs. It is the only model in this edition for which that holds.
- GLM 5.3 Flash, GPT-6 Luna, GPT-6 Sol, Haiku 5.5, MiniMax M3, MiMo v2.6 Pro, Opus 5, Opus 4.8, Opus 4.7 and Opus 4.6 are about as close as Sonnet 5.5. Each interval crosses zero. GLM 5.3 Flash comes nearest to a real difference (0.45 lower, interval −1.07 to +0.12).
- Every other model in the table agrees less closely than Sonnet 5.5. Each interval excludes zero. That includes both Sonnet 4.6 configurations: Sonnet 4.6 (production) is 2.15 points further on average (interval +0.96 to +3.46) and closer than Sonnet 5.5 on only 8 of 32 PRs.
Stability
Per-PR coefficient of variation over 3 runs, using the sample standard deviation; 32 PRs per row.
| Model | Average CV | Worst PR | PRs above 10% (of 32) |
|---|---|---|---|
| GPT-6 Astra | 3.8% | 14.7% | 3 |
| Fable 5 (reference) | 4.0% | 10.2% | 1 |
| Sonnet 4.6 (production) | 4.0% | 11.5% | 1 |
| Sonnet 5 | 4.1% | 10.2% | 1 |
| Opus 5.5 | 4.2% | 16.7% | 3 |
| Opus 4.8 | 4.5% | 21.7% | 4 |
| GPT-5.6 Terra | 4.7% | 14.2% | 2 |
| Kimi K2.6 | 4.7% | 43.3% | 3 |
| GPT-5.6 Sol | 4.9% | 20.4% | 4 |
| Sonnet 5.5 (default) | 5.1% | 13.3% | 4 |
| Opus 4.7 | 5.1% | 37.8% | 4 |
| Haiku 5.5 | 5.8% | 13.5% | 7 |
| MiniMax M3 | 5.9% | 24.7% | 6 |
| GPT-6 Sol | 6.0% | 20.4% | 4 |
| GPT-5.6 Luna | 6.1% | 26.7% | 7 |
| GLM 5.1 | 6.1% | 34.6% | 6 |
| GPT-6 Luna | 6.3% | 16.9% | 8 |
| Kimi K3 | 6.3% | 21.7% | 9 |
| MiMo v2.6 Pro | 6.5% | 22.2% | 10 |
| Opus 5 | 6.7% | 43.3% | 5 |
| Hunyuan 3 | 6.7% | 25.0% | 8 |
| Opus 4.6 | 7.3% | 24.1% | 9 |
| Sonnet 4.6 (thinking on) | 7.9% | 43.3% | 5 |
| GLM 5.3 Flash | 8.7% | 86.6% | 9 |
| GLM 5.3 | 9.8% | 43.3% | 11 |
| MiMo v2.6 Flash | 11.2% | 78.1% | 11 |
| DeepSeek V4.1 Flash | 13.9% | 57.7% | 11 |
| LongCat 2.0 | 18.4% | 61.9% | 21 |
Sonnet 4.6 (production) is steady partly because of how it is called: at temperature: 0 with thinking off it is close to deterministic. Sonnet 5.5 rejects temperature: 0, and production calls it with thinking on, as benchmarked here. Among the Anthropic models that think by default, average CV ranges from 4.0% (Fable 5) to 6.7% (Opus 5). GLM 5.3 Flash's worst PR is a tiny one that it scored 1, 1 and 4, where a few points is a large percentage. Most of its run-to-run variation is on small PRs.
Cost and speed
Uncached list-price cost per call at the rates above, from the measured token counts of all 96 calls per row. Production Anthropic scoring caches the shared prompt prefix, so real per-call costs are lower than these. Latency is the median of the 96 calls, from one client on October 1, 2026 (Haiku 5.5 on October 7, the low-cost OpenRouter models on October 15).
| Model | Avg input tokens | Avg output tokens | Cost per call | Per 1,000 PRs | Median latency |
|---|---|---|---|---|---|
| Hunyuan 3 | 16,089 | 1,412 | $0.0018 | $1.80 | 14.1s |
| DeepSeek V4.1 Flash | 16,303 | 2,142 | $0.0018 | $1.80 | 11.8s |
| MiMo v2.6 Flash | 16,264 | 1,587 | $0.0027 | $2.70 | 24.6s |
| GLM 5.3 Flash | 15,876 | 1,748 | $0.0033 | $3.30 | 8.9s |
| GPT-6 Luna | 15,904 | 3,786 | $0.0035 | $3.50 | 38.0s |
| Haiku 5.5 | 25,626 | 5,054 | $0.0051 | $5.10 | 22.3s |
| GPT-5.6 Luna | 15,904 | 2,255 | $0.0059 | $5.90 | 26.2s |
| LongCat 2.0 | 16,327 | 1,575 | $0.0068 | $6.80 | 27.3s |
| MiniMax M3 | 15,965 | 2,153 | $0.0074 | $7.40 | 17.3s |
| MiMo v2.6 Pro | 16,264 | 1,856 | $0.0087 | $8.70 | 42.5s |
| Kimi K2.6 | 15,779 | 2,057 | $0.0106 | $10.60 | 20.4s |
| GLM 5.1 | 15,873 | 4,176 | $0.0280 | $28.00 | 63.9s |
| Kimi K3 | 15,788 | 1,863 | $0.0291 | $29.10 | 36.5s |
| GLM 5.3 | 15,876 | 1,923 | $0.0307 | $30.70 | 17.0s |
| GPT-6 Sol | 15,904 | 1,737 | $0.0492 | $49.20 | 23.9s |
| GPT-5.6 Terra | 15,904 | 2,979 | $0.0676 | $67.60 | 38.1s |
| Sonnet 5.5 (default) | 25,626 | 4,356 | $0.0948 | $94.80 | 29.0s |
| Sonnet 4.6 (production) | 18,593 | 2,875 | $0.0989 | $98.90 | 57.0s |
| Sonnet 4.6 (thinking on) | 18,622 | 4,318 | $0.1206 | $120.60 | 81.1s |
| Sonnet 5 | 25,624 | 7,679 | $0.1280 | $128.00 | 73.4s |
| GPT-5.6 Sol | 15,904 | 3,512 | $0.1339 | $133.90 | 51.3s |
| Opus 4.6 | 18,593 | 2,130 | $0.1462 | $146.20 | 49.2s |
| Opus 4.7 | 25,629 | 2,848 | $0.1993 | $199.30 | 39.1s |
| Opus 4.8 | 25,624 | 2,848 | $0.1993 | $199.30 | 37.5s |
| Opus 5.5 | 25,626 | 4,926 | $0.2010 | $201.00 | 44.2s |
| GPT-6 Astra | 15,904 | 2,113 | $0.2647 | $264.70 | 44.0s |
| Opus 5 | 25,624 | 5,708 | $0.2708 | $270.80 | 66.7s |
| Fable 5 (reference) | 25,624 | 4,309 | $0.4717 | $471.70 | 54.5s |
Newer Anthropic models count about 38% more input tokens for the same prompt (about 25,600 against 18,600 here), so compare per-call cost, not per-token rates.
Changelog
Newest first, one entry per change. Each new benchmark edition replaces the Latest benchmark results above rather than adding to them.
- October 15, 2026: GLM 5.3 Flash made selectable. On your own OpenRouter key. About as close to the reference as the default at about 3% of its cost per call, but noisier run to run. It runs with minimal reasoning, because it refuses requests with reasoning off. Seven other low-cost OpenRouter models were measured against the same October 1 Fable 5 runs and declined: MiniMax M3, MiMo v2.6 Pro, MiMo v2.6 Flash, GLM 5.3, LongCat 2.0, Hunyuan 3 and DeepSeek V4.1 Flash.
- October 13, 2026. Reorganized this page so the method and the rules for model changes stand on their own. Added How we handle model changes.
- October 7, 2026: Haiku 5.5 made selectable. About as close to the reference as the default at about 1/19th of the cost per call, but noisier, and it scores large features higher. It runs at its default effort,
medium; higher effort settings were slower than the default without closing the gap on large PRs. - October 2, 2026: default moved from Sonnet 4.6 to Sonnet 5.5. The same work scores about 12–19% lower on Sonnet 5.5 (12.1% lower on 327 PRs rescored with both models; 19.1% on the benchmark corpus). Sonnet 4.6 ran about 16% above the Fable 5 reference and Sonnet 5.5 runs about 6% below it, so most of the step is the old default's high lean going away. Sonnet 5.5 failed one pre-registered gate, run-to-run stability (5.1% against 4.0%), and we overrode it for the accuracy gain. Sonnet 4.6 stays selectable.
- October 1, 2026: October 2026 edition. 19 configurations on the 32-PR corpus, the first edition on the corrected prompt and the first to benchmark Sonnet 4.6 the way production calls it (thinking off). GPT-6 Luna made selectable, overriding a stability gate for its cost. Sonnet 5 again not promoted. GPT-6 Astra and GPT-5.6 Sol declined.
- September 22–23, 2026. Opus 5.5 and the GPT-6 line benchmarked on the corrected prompt. Opus 5.5 and GPT-6 Sol made selectable.
- August 2026. 12 models on a 20-PR corpus. The reference moved from Opus 4.7 to Fable 5. The prompt did not yet supply effective-line counts, so these figures are not comparable with later editions. GLM 5.2, Mistral Small 4 and Mistral Medium 3.5 declined.