JMIR Publications

ABSTRACT

Background:

The past decade has seen artificial intelligence (AI) move from research curiosity to clinical challenge, with systems now matching or exceeding specialist performance on discrete diagnostic tasks. Yet for most health systems, this technical progress has translated into surprisingly little change at the bedside. Implementations stall, adoption remains uneven, and the gap between what AI can do in a paper and what it delivers in practice has become one of the more pressing questions in health informatics.

Objective:

This scoping review aimed to map the current evidence base for AI applications in healthcare quality and patient safety, identify patterns of clinical effectiveness and methodological limitations, and characterize structural barriers to implementation across four domains: diagnostic AI, predictive analytics, clinical decision support, and economic evaluation.

Methods:

A scoping review was conducted following the PRISMA extension for Scoping Reviews (PRISMA-ScR) and the Joanna Briggs Institute methodology. Systematic searches of three databases were performed covering January 2017 to March 2025. From 6,566 identified records, 80 peer-reviewed studies were included and synthesized using narrative synthesis. Study characteristics were mapped across domains, and methodological quality was assessed based on study design.

Results:

Included studies comprised diagnostic AI (n=28), predictive analytics (n=24), clinical decision support (n=18), and economic evaluations (n=10). Diagnostic AI demonstrated accuracies of 85–99% across imaging and pathology applications, with multiple RCTs confirming non-inferiority to specialist readers. Predictive analytics models reported mortality reductions of 12–58% for sepsis and deterioration; within the 24 predictive analytics studies included in this review, 79% (19/24) used retrospective designs, and a systematic review of machine learning for sepsis prediction reported that only a small minority of published models have undergone prospective external validation. Clinical decision support applications showed reductions in medication errors and alert fatigue, but algorithmic bias and limited generalizability were identified as recurring concerns. Economic evaluations reported returns on investment up to 451%; a systematic review of 66 health economic evaluations found that up to 91% failed to meet basic quality standards, frequently omitting implementation costs.

Conclusions:

This scoping review maps a consistent pattern across domains: strong technical performance in controlled settings is not reliably accompanied by prospective validation, equitable design, or transparent economic modeling. Three structural barriers were identified, including a validation gap, an implementation chasm, and infrastructure silos. These findings suggest that realizing the clinical value of healthcare AI requires a strategic shift toward rigorous prospective evaluation, standardized reporting, and equity-centered data ecosystems. Further research is needed to address generalizability and real-world implementation across diverse healthcare settings.