Warning: This is an author submission that is not peer-reviewed or edited. Preprints - unless they show as "accepted" - should not be relied on to guide clinical practice or health-related behavior and should not be reported in news media as established information.
Educational Quality in the Age of AI: Blinded Comparative Evaluation of ChatGPT, Gemini, and Claude on Educating Patients Newly Diagnosed with Kidney Cancer
ABSTRACT
Background:
A kidney cancer diagnosis is increasingly first interpreted not by a clinician but by a chatbot. As patients and trainees now learn from large language models (LLMs) before any consultation, the educational quality of these tools, and how it varies between them, matters to medical educators. Most prior kidney cancer work has examined a single platform and reported overall quality rather than characterizing how responses fall short.
Objective:
This study aimed to evaluate and compare the educational quality of responses generated by 3 widely used LLMs (ChatGPT, Gemini, and Claude) to patient-derived kidney cancer questions, and to classify the nature of their shortcomings rather than only their magnitude.
Methods:
A total of 15 patient-derived kidney cancer questions spanning 5 thematic domains were submitted to ChatGPT, Gemini, and Claude under an identical patient-perspective prompt. The 45 anonymized, randomly ordered responses were scored in a blinded consensus meeting by a urology panel using the Quality Analysis of Medical Artificial Intelligence (QAMAI) tool (6 items; total score 6-30). Models were compared with the Friedman test and Bonferroni-corrected Wilcoxon signed-rank tests. Beyond comparative scoring, a pre-specified deficiency taxonomy classified why responses lost marks, and the surgical (nephron-sparing) domain was appraised in depth as a worked educational example.
Results:
Educational quality was strongly model-dependent (Friedman P<.001). Mean total QAMAI scores were 23.3 for Claude (good quality; median 23, IQR 22.5-24), 21.9 for ChatGPT (moderate; median 22, IQR 21-24), and 16.6 for Gemini (poor; median 16, IQR 15.5-18). Claude and ChatGPT each significantly outperformed Gemini overall and on accuracy, relevance, completeness, and usefulness (all adjusted P<.05) but did not differ from each other. The deficiency taxonomy showed that Gemini’s failures were overwhelmingly omissions rather than errors: content was incomplete in all 15 (100%) responses and decision-relevant detail was missing in 11 (73%), versus 4 such instances for ChatGPT and none for Claude. No response from any model (0/45) cited a source. The hierarchy persisted across all 5 domains and in the surgical domain (Claude 24.5, ChatGPT 22.5, Gemini 18.0).
Conclusions:
The educational quality of LLM-generated kidney cancer information ranged from good to poor depending on the platform, and the dominant failure mode was silent omission rather than overt error, a pattern unlikely to alert a self-educating patient. LLMs should supplement, not replace, clinician-led education, and trainees should be taught to appraise artificial intelligence output critically, asking not only whether an answer is correct but what it has left out and whether it can be sourced.
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.