Real-SWE: even Claude Fable 5.1 fails 6 in 10 tasks on private code

If you have ever pointed an AI coding agent at your own repository after watching the same model breeze through a public benchmark, you already suspected something like this. Specific Labs' Real-SWE results confirm it with numbers: the environment where agents look brilliant is the environment they were trained on.

Public benchmarks were always a home game

The classic SWE-bench-style evaluation takes issues from popular open-source repositories. Those repositories live on GitHub, their code is public, and their issues and patches have been on the open web for years. Any model trained after a certain date has effectively seen the answers, the surrounding files, and often the fix itself.

Real-SWE inverts that setup. Its tasks come from private, proprietary enterprise codebases that were never published. There is nothing to memorise, no repository structure to recall from pretraining, no commit history to pattern-match against. What remains is the actual job: read unfamiliar code, understand what is being asked, change it correctly, and not break the rest of the system.

The depth of the tasks matters too. Real-SWE tasks touch a median of 11 files — nearly double the file spread of typical public benchmarks. That is not a single-function patch. That is the kind of change a senior engineer sketches out on a whiteboard first.

The leaderboard: nobody clears 40%

Running the strongest agents available in September 2026 against those private repositories produces a sobering table:

For context, the same generation of models scores 41.2% on SWE-Bench ProMax and 29.1% on Senior SWE-Bench — two other benchmarks built this year specifically because the old public sets had saturated. Real-SWE is not an outlier in exposing weakness; it is the most honest of the new batch, because its test set cannot leak.

The distribution is worse than the average suggests. Six out of ten sample tasks had a success rate below 15%. On one analytics stream reducer task, zero of 64 attempts produced a correct solution. This is not a leaderboard with a few easy tasks pulling the average up — it is a hard wall with a thin ramp in front of it.

How the agents actually fail

Three failure modes dominate, and anyone who reviews AI-generated pull requests will recognise all of them:

• Missed requirements. The agent solves a neighbouring problem that it understood better than the one it was given.
• Integration errors. The change works in isolation and breaks against the rest of the codebase — exactly what a 11-file median makes likely.
• Unverified assumptions. The model invents a function signature, an API contract, or a data shape, and never checks it against the repository it is editing.

None of these are exotic. They are the same errors a new contractor makes in their first week — except the contractor asks questions, and the agent does not.

What a fix actually costs

The most useful number in the Real-SWE data is economic: Anthropic's Fable 5.1 averaged $18 per task fix on the benchmark. Combine that with the 38.8% resolve rate and the arithmetic gets uncomfortable: $18 ÷ 0.388 ≈ $46 per successfully solved task — and that is token spend alone, before a human opens the diff.

Because the majority of attempts fail, you pay for failure twice: once in inference, once in the engineer's time to review a patch that has to be thrown away. Anyone budgeting an agent rollout on price-per-token alone is budgeting for the wrong thing.

Here is how the current field prices out per million tokens, for teams weighing that trade-off:

At an indicative 1 EUR ≈ 1.10 USD, the frontier tier lands around €9/€45 per million tokens — before VAT, which European business customers pay on top of US list prices.

Why this hits European teams harder

Real-SWE's core premise — private, proprietary code — is precisely the thing European companies cannot casually ship to a US-hosted API. With GDPR obligations on personal data that frequently sits inside test fixtures, logs and database schemas, and with the AI Act's general-purpose model obligations now in their binding enforcement phase rather than the voluntary-pilot era, "we sent the repo to a vendor and hoped" is no longer a defensible position in an audit.

That reframes the leaderboard. The question is not only which agent scores highest, but which agent you are allowed to point at your codebase. Mistral Medium 3.5 is the cheapest EU-hosted option in the table above. DeepSeek-V4.1-Flash and Meta's Muse Spark 1.3 ship open weights you can run on your own hardware, which solves residency by construction.

That is also why our own AI Arena benchmark rig exists: an RTX 5060 Ti with 16 GB of VRAM running local models via Ollama. It is enough to keep code on-premise, but 16 GB is a hard ceiling — it constrains model size and context length, and no local model on that rig is going to reproduce a 38.8% resolve rate on 11-file enterprise tasks. Sovereignty and capability are still two different budgets.

What to do with this on Monday

Build your own mini-eval. Take twenty closed tickets from the last six months, strip the fixes, and run your agent against them. That is a Real-SWE you can actually measure — and it costs nothing but a few evenings.

Then budget accordingly. Treat the agent as a fast first-draft generator with a roughly one-in-three hit rate on hard multi-file work, not as an autonomous committer. Keep merge rights human. And if review time is your bottleneck rather than token spend, the $46-per-successful-fix figure above is the number to put in front of whoever signs off on the tooling budget.