JMIR Publications

ABSTRACT

Background:

Despite high reported accuracy on clinical and evidence appraisal tasks, artificial intelligence (AI)–generated medical information may lack explicit support from source documents. This creates challenges for digital health practitioners regarding transparency, auditability, and trust when AI systems are used for evidence synthesis, guideline development, and clinical knowledge management. Large language models (LLMs) can generate fluent and seemingly correct outputs, but existing evaluations often rely on agreement with human judgments and do not directly assess whether AI-generated content is grounded in underlying evidence.

Objective:

To measure evidence support and hallucination in AI-generated medical information by quantifying the extent to which LLM-generated risk-of-bias assessments are supported by source clinical trial reports.

Methods:

We evaluated 3 LLMs (GPT-5, OpenAI o3-mini, and GPT-3.5) on risk-of-bias (RoB 2) assessment using 97 randomized controlled trials from a Cochrane systematic review with available human annotations and full-text reports. Model outputs were constrained to structured RoB 2 signaling questions and domain-level judgments. For each generated claim, relevant text passages were retrieved from trial reports using the Okapi BM25 algorithm. A verification step assigned evidence verdicts (Supported, Contradicted, Not Found, or Out of Scope) with minimal quotations. We quantified evidence support rates and conservative and strict hallucination rates. Task performance was evaluated using exact and binary accuracy, sensitivity, specificity, F1 score, Youden’s J, and agreement with human reviewers using Cohen’s κ and Fleiss’ κ.

Results:

Binary accuracy of AI-generated risk-of-bias judgments was high across domains (90%–98%), whereas exact accuracy was substantially lower (42%–71%), reflecting frequent disagreements in severity classification despite correct risk direction. GPT-5 achieved the strongest overall performance, including perfect binary accuracy for overall risk-of-bias conclusions and the highest agreement with human reviewers (quadratic κ up to 0.81). However, evidence support rates across models ranged from only 60% to 65%, with conservative hallucination rates of 34%–37%. GPT-5 showed the highest mean evidence support (64.3%) and the lowest strict hallucination rate (35.7%). Mean top-1 BM25 retrieval scores were similar across models (approximately 30–31), indicating that differences in hallucination were driven primarily by model reasoning rather than information retrieval.

Conclusions:

AI-generated medical information can achieve high decision-level accuracy while still lacking documentary support in a substantial proportion of outputs. Measuring evidence support and hallucination reveals important limitations that are not captured by agreement metrics alone. Retrieval-based evidence verification provides a reproducible and transparent approach for evaluating and improving the reliability of AI-generated medical information, with direct relevance to digital health practice, evidence-based medicine, and medical informatics. Clinical Trial: Not applicable.