JMIR Publications

ABSTRACT

Background:

Machine-learning models now equal or surpass expert performance on many circumscribed clinical tasks when measured on retrospective data, and the clinical decision support system (CDSS) is the channel through which such models are meant to reach the consultation. Primary care, which carries the largest and least differentiated stream of presentations under acute time pressure, is widely held to be where AI-CDSS should help most. Performing well on stored data, however, is a claim about a model, whereas improving care is a claim about a clinician and a patient, and the two are routinely conflated.

Objective:

To synthesise controlled, real-world evaluations of clinician-facing, AI-enabled CDSS in primary care, and to establish where along the causal chain, from diagnostic detection through clinician behaviour to patient-important outcomes, evidence of benefit holds and where it fails.

Methods:

Following PRISMA 2020, five databases (PubMed, Scopus, CINAHL, Web of Science, and the Cochrane Library) were searched from 2011 to 2025. Eligible studies evaluated a learned or adaptive model (machine learning, deep learning, natural language processing, or large language models) used by clinicians in routine primary or ambulatory care with an evaluable clinical, behavioural, or process outcome. Simulation, vignette, benchmark, model-development, prototype, and non-clinician studies were excluded. To avoid conflating learned models with deterministic logic, studies of rule-based or fixed-equation tools were analysed separately from genuinely AI-enabled studies. Records were screened in duplicate with adjudication. Risk of bias was assessed with design-specific tools (RoB 2, ROBINS-I, QUADAS-2), and certainty was rated with GRADE applied to bodies of evidence defined by outcome rather than to individual studies.

Results:

Of 1293 records identified and 680 screened, 18 studies were included; 4 used genuinely AI (learned) models and 14 used rule-based or fixed-equation logic. Among the AI-enabled studies, support increased diagnostic case-finding (new low ejection-fraction diagnosis 2.1% vs 1.6%; odds ratio [OR] 1.32, 95% CI 1.01-1.61), achieved high diagnostic accuracy (melanoma detection area under the curve 0.960), improved a care-process outcome (urinary tract infection treatment success 75% to 80%; P<.001), and reduced clinician workload (asthma record-review time 3.5 vs 11.3 minutes; P<.001). No AI-enabled study demonstrated a patient-important benefit; the single such primary endpoint, childhood asthma exacerbation, was unchanged (OR 0.82, 95% CI 0.37-1.96). The rule-based studies followed the same gradient, with process and surrogate gains but inconsistent or unmeasured patient-level effects. Certainty was moderate for diagnostic case-finding and low for accuracy, care-process, and patient-important outcomes.

Conclusions:

Evidence that AI decision support improves primary care concentrates at detection and process, thins at clinician behaviour, and is effectively absent at patient-important outcomes. The distance between decision-support and decision-impact is a socio-technical and evaluation problem rather than an algorithmic one, and future trials should be powered for patient-important endpoints rather than model performance. Clinical Trial: PROSPERO CRD420261302104; https://www.crd.york.ac.uk/PROSPERO/view/CRD420261302104