What the study ran
The paper, reported by Bioengineer.org on 21 November, used 262 exam-style questions split into 1,310 individual true/false items. Every model answered the full bank. Answers were first checked against the solutions printed in the book. Separately, three anaesthesiology specialists rated 495 explanations covering 165 questions – three models times 165 – without being told which model produced which text.
That design matters. Most published medical LLM evaluations only compare a model's answer with a published answer sheet and stop there. Here the team treated the book as a suspect, not as ground truth, and adjudicated disagreements against current clinical guidelines.
Three models, one result
On the book's own answers the spread was 89.9% to 91.3%. Paired differences stayed inside three percentage points, well within the pre-specified five-point equivalence margin, with p < 0.001. Statistically, the study found no ranking.
That is a large shift from the previous generation of results: earlier generations of these models often sat near the pass mark rather than well above it.
Once contested items were removed, accuracy rose to between 93.8% and 95.5%. Removing bad questions made all three models look better by roughly three to four points, which is a useful reminder about how much of an evaluation score is really a property of the test set.
The disagreements pointed at the book
This is the number worth carrying around. Of 81 items where the models did not agree with each other, guideline-based adjudication found the textbook answer incorrect or contested in 59 cases, or 72.8%. For comparison, a random sample of 20 agreed-upon items produced one contested answer.
The practical reading: when three independent frontier models converge on a different answer than the printed book, the disagreement is a weak signal about model quality and a strong signal about item quality. Anyone building a revision tool on top of a commercial question bank should expect this, especially in anaesthesia, where dosing thresholds, resuscitation algorithms and guideline versions move.
Explanations do not predict correct answers
The second half of the study is the more uncomfortable one for anyone who uses a model's reasoning as a sanity check. Three specialists scored explanations across several axes. Completeness had to be dropped because inter-rater agreement was too low (Gwet AC1 = 0.428, against a 0.60 threshold). The remaining axes showed no significant differences between models (p ≥ 0.305).
Worse for the "read the explanation and judge for yourself" workflow: explanation quality predicted answer correctness with an AUC-ROC of 0.622. Negative predictive value came in at 89.3% to 90.9% against a base rate of 88.5%. In plain terms, a fluent, complete-looking explanation gave the reader almost nothing beyond the prior probability that the answer was right anyway.
The failure mode is familiar from medical LLM evaluations: a model that is wrong sounds exactly like a model that is right. Fluency is cheap; verification is not.
What a full question bank costs
Running a bank this size is a rounding error on any cloud bill, so the price differences between the three models are academic for a single trainee. At scale, in a tool used by a residency programme, they are not. All figures below are list API prices in USD per million tokens. The last column combines one million input tokens and one million output tokens.
| Model | Input $/1M | Output $/1M | Combined $/1M+1M |
|---|---|---|---|
| Claude Sonnet 5 | $2.00 | $10.00 | $12.00 |
| GPT-5.6 Terra | $2.00 | $12.00 | $14.00 |
| Gemini 3.5 Flash | $0.75 | $4.50 | $5.25 |
Gemini 3.5 Flash comes in at less than half the combined cost of GPT-5.6 Terra ($5.25 vs $14.00) and below Claude Sonnet 5 ($12.00). Since the study found no significant accuracy difference between the three, price and latency are the only rational tie-breakers for this specific workload. EU business customers pay VAT on top of these rates, usually handled by reverse charge for B2B accounts, so the invoice you see is not the invoice a US customer sees.
European availability and the AI Act
All three are sold in the EU through the vendors' first-party APIs and through regional cloud endpoints, and all three handle English-language medical text without special configuration. Whether a specific model version is served from an EU region depends on the provider's rollout schedule, so if data residency matters for your deployment, check the region list for that model ID rather than for the vendor's product family.
Regulation is mostly a side issue here. A trainee using a chatbot for self-testing is not deploying a high-risk system. The AI Act's Annex III education category covers systems used to determine access to education, evaluate learning outcomes, assess the appropriate level of education, or monitor prohibited behaviour during tests. An institution that grades or admits people with a model falls inside that scope; a revision aid does not. Note that the deadline for standalone Annex III high-risk systems was pushed to 2 December 2027 under the Omnibus agreement, while general-purpose model obligations have been under full statutory enforcement by the European AI Office since 2 August 2026, with fines up to €15 million or 3% of global turnover.
Limits of the result
One commercial question book, one specialty, true/false format. The study measured agreement with exam-style answers, not clinical decisions, not patient outcomes and not learning retention. A trainee who works through 262 questions with a model learns something; whether that something survives to the oral exam is a different experiment and nobody has run it here. The authors also had only three raters, which is why the completeness axis had to be discarded.
What the data does support is narrow and useful: on this question bank, at this point in the model cycle, the choice of vendor no longer explains the errors. The questions do.
Can a frontier model replace a commercial question bank for EDAIC revision?
Not on this evidence. The study tested answer accuracy on questions taken from a question bank, not the model's ability to generate new items or to teach. A model that answers 90% of printed questions correctly can still write plausible new questions with wrong answers, and nothing in the design rules that out.
Why did the models agree with each other more than with the textbook?
Guideline adjudication found the textbook answer incorrect or contested in 59 of 81 items where the models disagreed with each other (72.8%), against one contested item in 20 random controls. The study does not establish why the models converged on those answers; it only compares their output with current guidelines and the printed key.
Does using AI for exam preparation fall under the EU AI Act's high-risk rules?
A trainee's own study tool does not. Annex III covers education systems used to determine access, evaluate learning outcomes, assess the appropriate level of education, or monitor prohibited behaviour during tests. Tools an institution uses to grade or admit students sit in that category; the compliance deadline for standalone Annex III systems is 2 December 2027.