108 psychiatrists, 10 models, and 300 questions: How the study was conducted

A research team from German hospitals led by Esra Lenz and Emanuel Schwarz created three realistic simulations of psychiatric interviews — with a patient suffering from depression, mania, and schizophrenia. All were performed by the same trained actor (standardized patient), the interviews were conducted by certified psychiatrists, and 108 clinicians (mostly residents with an average of 5.4 years of experience) evaluated 100 items according to the AMDP system — the European standard for documenting psychopathological findings.

In parallel, the same interviews in the form of text transcripts were evaluated by 10 large language models: GPT-5.1, Gemini-3-Pro-Preview, Claude-Sonnet-4.5, Gemini-2.5-Flash, Mistral-Large-3, Qwen3-Next-80B, DeepSeek-R1, Kimi-K2-Thinking, GPT-4o-mini, and GPT-OSS-20B. The models worked only with text — no image, no tone of voice, no body language. The doctors, on the other hand, had complete audiovisual recordings at their disposal. The reference standard was the consensus of three expert AMDP trainers.

The results were published on July 7, 2026 in the journal npj Digital Medicine, which falls under the Nature publishing group.

GPT-5.1 vs. junior psychiatrists: Results by diagnosis

The highest overall accuracy (accuracy 0.72) was achieved by GPT-5.1 and Gemini-3-Pro-Preview when using majority voting across three runs. This places them at the 64th percentile of the distribution of junior psychiatrists. In other words — GPT-5.1 outperformed 64% of doctors in this study.

However, the differences varied significantly by diagnosis:

• Depression: GPT-5.1 scored 0.81 (doctors: 0.79) — the best result, the model slightly above average
• Mania: GPT-5.1 scored 0.76 (doctors: 0.68) — the model significantly better, 93rd percentile
• Schizophrenia: GPT-5.1 scored 0.60 (doctors: 0.58) — the hardest category for both sides

Interestingly, even the doctors themselves showed considerable variability — their overall accuracy ranged from 0.36 to 0.89. Inter-rater agreement between two random psychiatrists reached only 65.4%. Psychiatric diagnosis is a subjective discipline, and that is precisely why AI can play an important role in it.

The GPT-OSS-20B model (an open-source variant of a smaller model) achieved an overall accuracy of just 0.40, while DeepSeek-R1 finished at 0.52. Proprietary models from OpenAI and Google thus clearly lead in clinical tasks for now.

AI and doctors make opposite errors — and that's good news

The study revealed fundamentally different error profiles between humans and machines. Doctors tended to over-infer the presence of symptoms (false positives) — of 69 errors, 44 consisted of marking an item as present when, according to the expert panel, it was not. In other words: doctors "saw symptoms even where there were none."

GPT-5.1, on the other hand, erred through conservatism — of 84 errors, in 28 cases it marked a symptom as "not assessable," even though according to experts it was not present. The model had a higher threshold for evidence and preferred not to assess a symptom at all if it lacked sufficient information.

This complementarity is the key finding. When the researchers simulated scenarios where two doctors disagreed (which occurred in 35.5% of cases), involving GPT-5.1 as a supervisor led to a significantly more accurate decision than random selection between disagreeing clinicians (p 0.0002). AI proved to be a better "referee" than a randomly chosen second doctor — and in the case of depression and mania, it even outperformed the supervision of a board-certified psychiatrist.

Why schizophrenia remains a tough nut to crack

The lowest accuracy for schizophrenia has a specific explanation. The AMDP system contains 9 items dependent on visual observation (e.g., motor restlessness, affective lability, mannerisms, parakinesis). These symptoms cannot be read from a text transcript — they must be seen. GPT-5.1 marked 63% of these as "not assessable," while doctors, who had video, almost never did so (3.9%).

When the researchers excluded these visually dependent items from the analysis, GPT-5.1's accuracy for schizophrenia rose from 0.60 to 0.66, for mania from 0.76 to 0.81, and for depression from 0.81 to 0.85. This suggests that multimodal models capable of processing video recordings as well could be even more accurate in psychiatric diagnosis.

What this means for psychiatry — and for the Czech Republic

The study comes at a time when both technology companies and academic institutions are investing significant resources in psychiatric AI. This April, PsychFound was published in Nature Machine Intelligence — a specialized 7-billion-parameter model that, in a real prospective study, demonstrated that residents using its assistance achieve higher quality consultations, more accurate diagnosis, and shorter documentation time. Also in April, PsychiatryBench was released, a benchmark with 5,188 expert-annotated items that confirmed that current LLMs achieve over 84% success in psychiatric reasoning.

For the Czech healthcare system, these findings have concrete implications. The Czech Republic has long struggled with a shortage of psychiatrists — according to ÚZIS data, some regions have fewer than 5 outpatient psychiatrists per 100,000 inhabitants. Waiting times for an initial examination routinely exceed 3 months. An AI assistant that could help with evaluating diagnostic interviews, screening, and proposing treatment plans could significantly relieve the overburdened system.

GPT-5.1 and Gemini-3-Pro-Preview are of course not medical devices — nor do they have certification under the European EU AI Act, which classifies AI systems in healthcare as high-risk. This means that before clinical deployment, they would need to undergo rigorous certification. However, the study shows the direction: AI doesn't need to replace doctors. It just needs to do what it does well — be a conservative, systematic, and tireless check on human intuition.

AI in psychiatry: assistant, not replacement

The study authors themselves emphasize that this is a proof-of-concept — a demonstration of the concept on three simulated interviews, which cannot be generalized to all clinical practice. Real deployment will require validation on real patients, larger samples, and multimodal input. Even so, there are already concrete scenarios where LLMs could help right away:

• Second reading: AI reads the interview transcript and highlights symptoms the doctor may have missed
• Standardization: the model applies the same criteria to every patient, regardless of fatigue or time of day
• Education: junior psychiatrists can practice diagnostics on AI — the model shows where they differ from expert consensus
• Decision-making in disagreement: in cases where two doctors disagree, AI can offer a third, conservative perspective

GPT-5.1 and Gemini-3-Pro-Preview are models also available in Czech — both understand the Czech language at a very high level. This means that once certified psychiatric tools built on these models become available, Czech language will not be a barrier.