JMIR Publications

ABSTRACT

Background:

While systematic reviews can identify reported results of artificial intelligence/machine learning (AI/ML) research, they only represent the numerator of the total research conducted.

Objective:

To estimate the proportions and patterns of result disclosures of AI/ML research on ClinicalTrials.gov and peer-reviewed publications indexed in PubMed or Scopus, and to reveal the extent of undisclosed results years after study completion.

Methods:

Using data from the Clinical Trials Transformation Initiative Aggregate Analysis of ClinicalTrials.gov, we identified studies that started and completed between January 2010 to December 2023 and contained AI/ML-specific terms in study descriptions, official title, brief summary, interventions, conditions, detailed descriptions, or sponsors’ keywords. For 842 completed studies, we searched PubMed and Scopus for publications that included study identifiers and AI/ML-specific terms within the titles or abstracts. We calculated disclosure rates within 3 years of study completion and the median time to disclosure from the ‘primary completion date’ to the ‘results first posted date’ on ClinicalTrial.gov or the earliest date of journal publication.

Results:

Of 842 completed studies (357 interventional; 485 observational), 5.5% (46/842) disclosed results on ClinicalTrials.gov, 13.3% (112/842) in journal publications, and 17.2% (145/842) through either dissemination route within 3 years of completion. The disclosure rates were higher among trials: 10.4% (37/357) on ClinicalTrials.gov, 18.8% (67/357) in journal publications, and 25.8% (92/357) through either route. Rates were also higher among randomized controlled trials: 11.3% (23/203) on ClinicalTrials.gov, 23.6% (48/203) in journal publications, and 31.5% (64/203) through either route. Nevertheless, most study findings (82.8%; 697/842) remained undisclosed after 3 years of completion. Trials employing randomization (vs. nonrandomized) or masking (vs. open-label) had higher disclosure rates and shorter times to disclosure. Although 85% (305/357) of completed trials had sample sizes of ≤1000, larger trials (n>1000) had higher publication rates (28.8%; 15/52) than smaller trials (n≤1000) (17.0%; 52/305). Conversely, large observational studies (n>1000) published less (6.7%; 9/134) than smaller studies (10.3%; 36/351). The most published sponsors were academia (14.7%; 38/259) and hospitals (12.4%; 42/340), followed by industry (11.0%; 16/146). Although high-income countries accounted for 81.9% (86/105) of published studies, the publication rates per completed study did not greatly differ among high-income (14.0%; 86/613), upper-middle (11.5%; 16/139), and lower-middle countries (13.0%; 3/23). Of studies with disclosed results, the median times to report through ClinicalTrials.gov and in journal publications were 505 days (IQR: 399-676) and 411 days (IQR: 274-676), respectively. Single-center studies were more common than multicenter studies for both trials (83.3%; 290/348) and observational studies (82.3%; 377/458).

Conclusions:

Of registered AI/ML studies completed during 2010-2023, over 80% of the findings remained undisclosed three years after study completion. While methodological rigor was generally associated with higher publication rates, a predominance of single-center designs was notable in both interventional and observational studies.