JMIR Publications

ABSTRACT

Background:

LLM-based AI teaching agents are increasingly adopted in medical education, yet pedagogical quality is typically assessed through platform-generated scores that lack standardization and may not reflect actual teaching effectiveness.

Objective:

This study aimed to develop and validate a multidimensional evaluation framework for AI teaching agents and to examine how role-play configuration, content domain, and learner gender influence teaching quality in a medical education context.

Methods:

Eight AI teaching agents covering an endocrinology curriculum were deployed across four role-play paradigms (Patient, Student, Expert, and Family). Twenty-two fourth-year medical students generated 175 dialogues, evaluated by both platform scoring and an independently applied eight-dimension rubric (knowledge accuracy, pedagogical guidance, knowledge coverage, role-play quality, difficulty calibration, medical safety, student engagement, and feedback quality; 100 points total).

Results:

Platform and rubric rankings were substantially discordant (Spearman ρ = −0.405); the platform's third-ranked agent scored last on independent evaluation. Knowledge accuracy was uniformly high (CV = 6.4%), while adaptive difficulty calibration and formative feedback remained consistently suboptimal (attainment: 55% and 45%). Role-play configuration significantly influenced all eight dimensions (Kruskal-Wallis, all P < 0.001). Role-play quality and knowledge coverage were inversely correlated at the dialogue level (ρ = −0.477, P < 0.001), confirming a structural trade-off between emotional engagement and knowledge delivery. Content domain modulated four dimensions independently; student gender had no effect.

Conclusions:

Platform-generated scores systematically misrank AI teaching agents and should not be used as the sole quality indicator. The proposed eight-dimension rubric offers a standardized diagnostic alternative, and the empathy–knowledge trade-off demonstrates that multi-role complementary deployment is necessary to optimize both engagement and knowledge delivery.