How the study was conducted

A research team led by Yousef Mohammed designed a rigorous two-part benchmark. The first part consisted of 70 multiple-choice questions covering three key areas: clinical oncology, radiation physics, and radiobiology. The second, more challenging part included 25 open-ended questions from clinical practice — the type of questions where there is no single correct answer and where expert judgment based on experience is required.

Responses to the open-ended questions were not evaluated by an algorithm but by three independent radiation oncologists using a five-point Likert scale. They assessed two dimensions: correctness (how factually accurate the answer is) and usefulness (how much the answer would actually help in clinical practice). A mixed-effects model was used for statistical evaluation.

Gemini narrowly wins the knowledge test

In the multiple-choice section, ChatGPT (GPT-5.2) achieved an accuracy of 94.3%, while Gemini scored 97.1%. The three-percentage-point difference may not be dramatic, but it shows that Google has significantly closed the gap with OpenAI in domain-specific knowledge over recent months.

The results from the open-ended questions are more interesting. Here the differences virtually vanish: ChatGPT achieved an average correctness score of 4.71 out of 5, Gemini 4.67. For usefulness, it was 4.63 for ChatGPT and 4.64 for Gemini. In other words — both models are virtually equal in more challenging clinical tasks and their performance cannot be statistically distinguished.

Where the models still have room for improvement

The study doesn't just look for a winner — the analysis of limitations is more important. The researchers identified several areas where both models fall short:

Complex clinical scenarios requiring prioritization. When the model had to decide which patient to prioritize with limited radiation treatment capacity, answers were often too general or lacked specific clinical justification.

Individualized decision-making. The models reproduce textbook knowledge well but fail when specific circumstances need to be taken into account — the patient's age, comorbidities, previous treatment, or patient preferences.

Minor discrepancies with current clinical evidence. The authors noted cases where model responses diverged from the latest clinical guidelines. This is a typical LLM issue — training data has a cutoff date and the models do not have access to the most recent research.

Broader context: AI in medicine

This study fits into the growing wave of research focused on the use of language models in healthcare. It's worth recalling that as early as 2023, GPT-4 passed the USMLE medical licensing exams with scores in the top decile. Google also introduced MedLM this year — a suite of models specialized specifically for medical tasks, built on the Gemini architecture.

Radiation oncology is a specific discipline, however. It combines deep knowledge of ionizing radiation physics, tumor biology, and clinical decision-making. The fact that both models achieve over 90% accuracy in this field is remarkable — especially considering that they were not specialized in radiation oncology.

For comparison: similar studies in other specialties show a spread from 60% to 95% depending on question difficulty. Results of 94–97% in radiation oncology are among the best we have seen so far.

What this means for Czech healthcare

There are approximately 350 radiation oncologists working in the Czech Republic, and over 90,000 new cancer cases are diagnosed each year. Any tool that could help doctors with education, exam preparation, or quickly navigating the scientific literature has the potential to improve the quality of care.

However, it's important to be realistic. Neither ChatGPT nor Gemini is certified as a medical device under the European Medical Device Regulation (MDR). The EU AI Act, which entered into force in 2024 and is being phased in, also classifies AI systems used in diagnostics and treatment as high-risk. This means strict requirements for transparency, validation, and human oversight.

In practice, this means that before models like ChatGPT or Gemini can reach Czech hospital departments as official support tools, there is still a long road ahead — clinical trials, certification, and regulatory approval.

Who can benefit from this already

Even without official certification, language models are finding use in medical education. Medical students and junior doctors use them to prepare for board exams. Researchers use them to quickly search the scientific literature and summarize studies. And patients — although official guidelines often discourage it — turn to AI with questions about their diagnosis.

It is precisely for this last group that the study sends an important signal: models are increasingly accurate, but they still do not replace professional medical judgment. The study authors put it clearly: "Further studies are needed to determine whether this performance translates into meaningful clinical benefit in real-world radiation oncology practice."

ChatGPT vs. Gemini: a comparison for non-experts

For the average user looking to choose between the two models, it makes sense to know the practical differences. ChatGPT (GPT-5.2) is available in a free version with limited capacity and in a paid ChatGPT Plus version (approximately $20/month). It supports Czech and can communicate in natural Czech language. Gemini is integrated into the Google ecosystem — available via the web, mobile app, and in the Advanced version with a Google One AI Premium subscription (approximately €22/month). It also supports Czech at a good level.

Both models handle Czech decently, but for professional medical communication — especially when you need to consult technical terms — English remains significantly more reliable. Neither model has an official Czech localization for medical use.