Accepted for/Published in: Journal of Medical Internet Research
Date Submitted: Mar 3, 2026
Date Accepted: Sep 14, 2026
Reporting Gaps and Challenges in Randomized Trials of Generative AI for Medical Use: Insights from a Systematic Review
ABSTRACT
Background:
Generative artificial intelligence (Gen AI) is increasingly applied in medicine, and evidence of its clinical safety and efficacy requires high-quality reporting in randomized controlled trials (RCTs). The CONSORT-AI extension provides guidance for AI trials, but its suitability for Gen AI remains unexamined.
Objective:
This systematic review aimed to evaluate the adherence of medical Gen AI RCTs to the CONSORT-AI checklist, identifying existing reporting gaps and the unique challenges posed by the characteristics of Gen AI.
Methods:
A systematic search was conducted through PubMed, CENTRAL, and Embase for RCTs published up to April 16, 2025, involving Gen AI for medical purposes. Two reviewers independently assessed item-level adherence for 13 included studies. Adherence was summarized as the proportion of studies categorized as "Reported" for each checklist item.
Results:
All 13 studies utilized Large Language Models, yet none explicitly referenced or applied CONSORT-AI. While reporting was complete for Title/Abstract and Introduction items, it was markedly lower for Methods and Results domains. Poorly reported items included input data criteria, trial integration, algorithm version, handling of poor-quality data, and performance-error analysis.
Conclusions:
Current reporting practices for Gen AI RCTs show a substantial discrepancy with CONSORT-AI requirements, particularly regarding prompt management, potential harms, and data privacy. There is an urgent need to refine reporting standards to address the interactive nature of Gen AI, ensuring reproducibility, transparency, and clinical safety in future medical applications. Clinical Trial: PROSPERO CRD42051034690; https://www.crd.york.ac.uk/PROSPERO/view/CRD420251034690
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.