JMIR Publications

ABSTRACT

Background:

Health professions education faces increasing challenges from rising healthcare complexity, pedagogical shifts, and constrained curricular space, alongside rapidly expanding knowledge and technological advances. While artificial intelligence (AI) holds immense promise for transforming health professions education, evidence of its effectiveness remains unclear.

Objective:

We synthesized evidence from randomized controlled trials (RCTs) on the effectiveness of AI in undergraduate health professions education in improving learning outcomes.

Methods:

We searched PubMed and Cochrane (which covered PubMed, Embase, CINAHL and trial registries) from database inception till 19 November 2025 for RCTs that compared AI against standard educational interventions. We categorized outcomes according to Kirkpatrick’s levels (reaction, knowledge, behavior and results), assessed risk-of-bias using the ROBUST-RCT tool, performed random-effects meta-analysis (RevMan 5.4) and rated certainty-of-evidence using the GRADE approach.

Results:

Of 19303 unique records identified, 50 RCTs (n=3,746 participants) published between 2020 and 2025 were included. The overall risk of bias was high in majority of the studies due to poor allocation concealment and blinding, and certainty of evidence ranged from low to very low. Students who received AI-assisted learning appeared to perform better in theoretical knowledge (standardized mean difference [SMD] 0.65, 95% CI 0.37–0.93, 20 studies, 1647 participants, I2=86%, low-certainty) and may have a positive effects on practical and personal skills (Practical: SMD 0.45, 95% CI -0.20–1.09, 6 studies, 449 participants, I2=89%; Personal: SMD 0.54, 95% CI 0.28–0.81, 5 studies, 420 participants, I2=36%; low-certainty), but effects on other learning outcomes are uncertain (very-low-certainty-evidence), including self-efficacy (SMD 0.94, 95% CI 0.56–1.33, 13 studies, 1020 participants, I2=87%), satisfaction (SMD 0.69, 95% CI 0.35–1.03, 17 studies, 1409 participants, I2=88%), clinical skills (SMD 0.78, 95% CI 0.35–1.21, 17 studies, 1235 participants, I2=92%) and task efficiency (SMD -0.10, 95% CI -1.89–1.68, 4 studies, 243 participants, I2=96%).

Conclusions:

In undergraduate health professions education, low-certainty evidence suggests that AI may improve some learning outcomes, including knowledge, personal and practical skills with unclear effects on others. However, substantial variation in study findings lowered our confidence on the estimates and no studies assessed higher-level outcomes of behavior and health outcomes. With the rising interest in AI, further RCTs are expected to provide updated results and strengthen the evidence base to inform educational practice. Clinical Trial: PROSPERO (CRD42021243832).