JMIR Publications

ABSTRACT

Background:

The rapid growth of research in artificial intelligence (AI) and machine learning (ML) continues. However, it is unclear whether this reflects an increase in desirable study attributes or perpetuates the same issues previously raised in the literature.

Objective:

To assess temporal trends in AI/ML studies over time and identify variations that are not apparent from aggregated totals at a single point in time.

Methods:

We identified AI/ML studies registered on ClinicalTrials.gov that started from 1 January 2010 to 31 December 2023. Studies were selected if any of the following terms were in the official title, detailed descriptions, brief summary, interventions, primary outcome, and sponsors’ keywords: AI-based, artificial intelligence, machine learning, deep learning, artificial neural network, convolutional neural network, deep neural network, multilayer perceptron, Bayesian network, naïve Bayes, classification tree, gradient boosting, k nearest neighbor, random forest, regression tree, vector machine, or natural language processing.

Results:

Of 3106 AI/ML studies, 842 studies were completed, and only 7.6% (235) were FDA-regulated. The most common study characteristics were: randomized trials (56.2%; 670/1193 interventional) and prospective designs (58.9%; 1126/1913 observational); focused on diagnostic (28.2%; 335/1190) and treatment (24.4%; 290/1190); sponsored by hospital/clinic (44.2%; 1373/3106), academia (28.0%; 869/3106); neoplasms (12.9%; 420/3245), nervous system (12.2%; 395/3245), cardiovascular (11.1%; 356/3245), and pathological conditions (10.0%; 325/3245) (multiple counts per study possible). Enrollment data was skewed to the right (maximum 13,977,257; mean 16,962; median 255). The most common size category was 101-1000 (44.8%; 1372/3061 excluding withdrawn/missing), but large studies (>1000) represented 24.1% (738/3061) of all studies, 29.0% (551/1898) of observational studies, and 16.1% (187/1163) of trials. Study locations were predominantly in high-income countries (75.3%; 2340/3106) followed by upper-middle-income (21.7%; 675/3106), lower-middle-income (2.8%; 88/3106), and low-income countries (0.1%; 3/3106). The fastest-growing characteristics over time were high-income countries (location); Europe, Asia, and North America (location); diagnostic and treatment (primary purpose); hospital/clinic and academia (lead sponsor); randomized and prospective designs; and sample sizes of the 1-100 and 101-1000 categories. Of completed studies, the availability of results posted on ClinicalTrials.gov was limited, ranging from 5.3% to 7.3%. Thus, the number of completed studies without posted results also increased, not only newly initiated studies.

Conclusions:

Much of the rapid growth in AI/ML studies comes from high-income countries in high-resource settings, albeit with a modest increase in upper-middle-income countries (mostly China). Lower-middle/low-income countries remain poorly represented. The increase in randomized or prospective designs, along with 738 large studies (n>1000), mostly ongoing, may indicate that enough studies are shifting from the in-silico evaluation stage towards stronger evidence generation. However, the ongoing limited availability of basic results posted on ClinicalTrials.gov contrasts with the rapid advancements in this field and the public registry’s role in reducing publication and outcome reporting biases.