Why GPT-5 and Claude stumble in the hospital

Brain scans from magnetic resonance imaging and computed tomography practically do not exist on the public internet. The reason is simple — they contain identifiable facial features of patients, and therefore are subject to strict legal protection. GPT-5, Claude Sonnet 4.5, and other frontier models thus trained on billions of documents, images, and conversations from the web — but not on what a real brain tumor, hemorrhage, or ischemia looks like in actual clinical data.

"Frontier models know the map, but not the terrain," summarize the authors of the study from the University of Michigan. Large language models have read anatomy textbooks and scientific articles, but they have never seen 566,915 consecutive clinical studies spanning trauma, tumors, infections, and inflammation that a functioning academic hospital accumulates over 20 years.

Vol-JEPA: How to teach AI to see a brain without a single label

The team's key innovation is the Vol-JEPA (Volumetric Joint-Embedding Predictive Architecture) architecture. It extends the JEPA family originally designed by Meta's chief AI scientist Yann LeCun for regular images and videos. The Michigan team adapted it for the first time to full three-dimensional medical volumes.

The principle is elegant: the model receives a small visible crop of a 3D scan and must predict what lies in the masked portion — not in pixels, but in an abstract latent space. This means it learns to recognize anatomically meaningful structures, not texture or noise. The method is entirely self-supervised — it needs no manual labels, no radiology reports, no pairing with text.

Training the entire model consumed fewer than 1,000 GPU hours on NVIDIA L40S cards — computationally manageable for any larger academic medical center. This means that Czech university hospitals, such as Motol or FN Brno, could theoretically train their own version on their archives.

21 points more accurate, half the errors

In a laboratory test on 300 expertly verified studies, NeuroVFM achieved a score in recognizing finding severity (normal, routine, urgent) that was 11 points higher than GPT-5 and 20.3 points higher than Claude Sonnet 4.5.

But the error rate is even more important. In generated radiology reports, the rate of critical errors was half that of GPT-5 (10% vs. 20%). NeuroVFM also far less frequently confused the left and right hemispheres — so-called lateralization errors, which are among the most dangerous failures in radiology, as they can lead to surgery on the wrong side of the brain.

Three blinded clinical experts preferred reports from NeuroVFM over GPT-5 at a ratio of more than 2:1.

A week in live operation on 1,155 patients

The most valuable part of the study was a prospective test in real hospital operation. From January 18 to 25, 2026, NeuroVFM-LLaVA — a system connecting NeuroVFM with the open-source language model Qwen3-14B — processed a total of 1,155 studies (601 MRI, 544 CT) that were generated during routine clinical care at that time. The model ran quietly in the background, without influencing actual patient treatment.

The results showed 92.6% balanced accuracy in detecting urgent findings according to the American Society of Neuroradiology definition, while GPT-5 achieved only 71.2% (P < 0.0001). Of the 134 patients with truly urgent findings that the model flagged, all were flagged correctly.

There is, however, a major "but": the model missed 21 of 155 patients with critical findings — overall sensitivity is 86.5%. That is significantly better than GPT-5, but far from sufficient for autonomous screening without radiologist oversight. Missing one in seven urgent patients would be clinically unacceptable. The authors openly acknowledge this: NeuroVFM is a research tool and is not FDA-approved for clinical use.

Anatomy emerged on its own

One of the study's most fascinating discoveries concerns what the model learned without explicit instruction. When the researchers visualized NeuroVFM's internal representations, they found that they spontaneously organized into neuroanatomically meaningful clusters — gray matter separate from white matter, cerebellum separate from cortex. And all of this without a single segmentation mask or anatomical label during training.

The model also proved modality-agnostic — a diagnostic classifier trained on CT scans worked almost equally well on MRI, despite these being physically entirely different imaging methods (one measures X-ray absorption, the other magnetic resonance of hydrogen nuclei). This capability did not appear in any of the alternative training methods.

"Health system learning": A new approach to clinical AI

The Michigan team introduces a concept they call "health system learning" — learning directly from the data that hospitals generate during routine operation. This is a fundamentally different approach from training models on internet data.

"Health systems are knowledge bases and data engines," the authors write. Instead of AI learning from descriptions of the clinical world on the internet, it should learn directly from the raw data that clinical operation produces. And this data already sits in the picture archiving and communication systems (PACS) of every larger hospital.

If the "health system learning" concept is confirmed by independent replication at other institutions, it could represent a scalable path to building specialized AI models across medicine — without the need for exceptionally expensive manual data annotation.

What this means for Europe and Czechia

From a European perspective, it is crucial that NeuroVFM is fully open-source. The code is available under the MIT license on GitHub and the trained model weights under the CC-BY-NC-SA 4.0 license on Hugging Face.

For Czech hospitals, this means the possibility of localizing and adapting the model to their own data and their own workflows — without dependency on American commercial providers. Given that the European AI Act classifies medical AI as high-risk, a transparent architecture with publicly available code is a significant advantage for the certification process. Czech university hospitals possess extensive imaging archives stretching back many years — precisely the type of data on which NeuroVFM demonstrates its capabilities.

It must be emphasized, however, that any clinical deployment would require prospective validation under the supervision of local regulatory bodies — in the Czech context, the State Institute for Drug Control. The model remains purely a research tool for now.

Specialist versus generalist

The Michigan study demonstrates a broader principle with implications far beyond radiology: general-purpose models have structural blind spots where they lack training data. GPT-5 can perfectly discuss brain anatomy because it has read hundreds of textbooks and articles about it. But when it needs to recognize a hemorrhage on a blurry CT scan from a night shift in the ER, it fails — because such data simply does not exist on the internet.

The authors put it concisely: "Frontier models know the map; health system learners know the terrain."

The team's long-term vision is not to replace GPT-5 and similar models, but to connect them: NeuroVFM as a specialized perception module providing clinically grounded interpretation of imaging data, and GPT-5 as a general reasoning module that thinks over that data. In the study, this already worked — GPT-5 served as a component for assessing the severity of findings generated by NeuroVFM.