How health information quality was measured

Researchers led by Dr. Kavita Jain compared three sources of information: traditional Google search, the freely available ChatGPT-4 Omni from OpenAI, and OpenEvidence, a specialized medical LLM with restricted access, intended exclusively for healthcare professionals. For each of 13 benign, precancerous, and malignant breast pathologies — from cysts to invasive carcinoma — evaluators asked standardized questions and assessed the answers using the DISCERN tool. DISCERN is an internationally validated tool developed by Oxford University that rates the quality and reliability of health information on a scale of 0–80 points. It is used by regulatory bodies, patient organizations, and libraries worldwide. It examines, for example, whether sources are cited, whether the text contains bias, whether it describes treatment risks, and whether the language is balanced — without exaggerated promises. In other words, precisely what a patient cannot assess in a doctor's office, but what can fundamentally influence their decisions.

What they found: Google at the bottom

The average overall DISCERN score for Google reached 35.9 points out of 80. ChatGPT scored 62.2 points and OpenEvidence 64.8 points. Statistical significance was undisputed — one-way ANOVA showed p 0.0001 and Tukey's post hoc test confirmed that both LLM models outperformed Google with p 0.001. The most interesting finding? There was no statistically significant difference between ChatGPT and OpenEvidence. This means that a freely available chatbot provides health information about breast pathology at a level comparable to a specialized medical model that the average patient has limited access to. They also evaluated individual components separately: - Reliability: Google 19.9 / ChatGPT 28.4 / OpenEvidence 28.3 - Quality: Google 13.2 / ChatGPT 30.5 / OpenEvidence 31.1 In both dimensions, LLM models overwhelmingly won. According to the authors, Google particularly failed in citing sources, information balance, and clearly separating verified facts from opinions.

Why Google has a problem with medicine

Traditional Google search suffers from a fundamental limitation: the algorithm does not sort results by professional quality, but by relevance, domain authority, and many other signals that have nothing to do with medical accuracy. The first two pages of results for the query "breast lump" today contain a mix of hospital websites, alternative medicine blogs, clinic PR articles, and ads for dietary supplements. For a layperson, it is practically impossible to distinguish which source to trust. Large language models work differently. Neither ChatGPT nor OpenEvidence selects from existing websites — they generate an answer based on patterns learned from vast amounts of text, including medical publications. The result is a coherent, understandable text that can also translate professional terminology into everyday language — which is crucial for patients. The study authors explicitly state that LLM models provide information that is "more accurate, higher quality, and does not overwhelm the patient with unnecessary medical jargon."

Google isn't sitting still — AI Overviews are changing the game

It's fair to add that Google is responding to this trend. Since mid-2025, it has been gradually expanding the AI Overviews feature — AI-generated summaries displayed above traditional search results. These summaries no longer suffer from the problem of disparate sources, as they synthesize an answer from multiple trustworthy websites at once. However, the CancerNetwork study compared traditional search without AI elements — precisely the version that most patients still use. From the Czech user's perspective, the situation is specific: Google AI Overviews have been available in the Czech Republic since spring 2026, but only in Czech and with limited quality compared to the English version. In contrast, ChatGPT handles Czech very well — GPT-4 Omni and newer models understand Czech queries and generate answers that hold up even compared to the English original.

Should you use ChatGPT as a doctor?

The short answer is: no. The long answer: as a tool for basic orientation yes — but always with verification. The study shows that LLM models can very effectively explain what a given diagnosis means, what treatment options are available, and what to prepare for. But what they cannot do — and can never replace — is individual diagnosis. ChatGPT cannot see your scans, does not know your medical history, and lacks the context of your overall health condition. Even with a DISCERN score of 62 out of 80, there is still room for inaccuracies. In the Czech context, it is important to know that no AI chatbot currently has certification as a medical device in the EU. Under the European AI Act (effective from February 2025, with phased implementation through August 2026), AI systems used for diagnosis and patient triage fall into the "high risk" category. Neither ChatGPT nor OpenEvidence have such certifications — and that is a good thing, because patient queries to a chatbot do not meet the definition of regulated healthcare AI. What to watch out for: - Hallucinations — LLM models occasionally generate convincingly sounding but factually incorrect information - Outdated knowledge — training data ends at a certain date; the model may not know newer treatment procedures - Missing context — every breast cancer case is individual; the model works with general information

Broader context: AI as a tool for health literacy

This study is not unique. In May 2026, the prestigious journal Nature published a study showing that general large language models (GPT-5, Claude) outperform specialized clinical AI tools on medical benchmarks. In July 2025, Radiology Business reported that LLMs helped patients better understand radiology reports — texts that are traditionally incomprehensible to laypeople. The trend is clear: AI chatbots are becoming a bridge between professional medicine and patient reality. Not because they replace doctors, but because they can translate "ductal carcinoma in situ" into the sentence "this is a non-invasive form of breast cancer that remains inside the milk ducts and does not extend into surrounding tissue." In the Czech Republic and across the EU, the question is emerging whether health authorities — such as SÚKL — should issue recommendations for patients on how to safely use AI tools for health information. In the US, some hospitals (including those whose doctors participated in the study) are already actively integrating LLMs into patient portals as a layer for explaining diagnoses. In Europe, similar initiatives are still lacking — with the exception of pilot projects in Germany and the Netherlands.

What to take away from this

It's not about whether AI chatbots will replace Google — they are already winning that battle in the field of health information. The question is: how quickly and how safely to integrate LLMs into patient education, so that patients receive accurate information while not falling under the impression that a chatbot replaces a visit to a breast specialist. The researchers conclude by recommending linking AI tools with validated assessment frameworks like DISCERN, and creating accessible educational materials that help patients understand their diagnosis without unnecessary fear — but also without false certainty.