JMIR Publications

ABSTRACT

Background:

Large language model (LLM)–based virtual-patient systems can score history-taking practice and provide immediate feedback, yet whether these scores align with performance on a platform-independent clinical examination remains incompletely tested. Most evaluations report internal platform metrics or simulated-patient fidelity, leaving criterion-related evidence limited.

Objective:

To test whether mean pre-examination AI-trainer practice scores were associated with OSCE history-taking scores, whether the association extended beyond complaint-matched rehearsal, and whether a model including the trainer score retained predictive performance in held-out students.

Methods:

At one medical school, eligible second-year students received a required six-case assignment on the AI-Powered Medical History-Taking Training and Evaluation System (AMTES), an LLM virtual-patient trainer, and subsequently completed an end-of-course OSCE. The four OSCE diagnoses differed from the training diagnoses: three shared a presenting-complaint category, but not a diagnosis, with an AMTES case; one OSCE case had no presenting-complaint match. Platform-to-OSCE record linkage was 474/480 (98.8%); 473 students with analyzable pre-examination practice formed the analytic sample. The primary predictor was each student’s mean normalized platform practice score over practices completed before their own examination. Analyses included Spearman correlations, ordinary least squares regression with OSCE-case fixed effects, leave-matched-case-out analysis, and 5-fold cross-validation. The observational design did not estimate causal effects.

Results:

OSCE scores (mean 81.5, SD 6.7) were associated with the mean platform practice score (Spearman ρ=0.388; P<.001). In the OSCE-case-fixed-effects model, each 1-SD higher platform score was associated with a 2.83-point higher OSCE score (95% CI 2.26–3.39). The association was observed in the low-back/flank-pain stratum, whose presenting complaint had no match among assigned AMTES cases (ρ=0.370, between students), and for a leave-matched-case-out score (ρ=0.398; n=339). Platform-score-only 5-fold cross-validation yielded Spearman ρ=0.37 and R²=0.17; adding the score to case fixed effects increased R² from 0.15 to 0.33 (joint-model Spearman ρ=0.57, conditional on assigned case). The association persisted after post hoc adjustment for a prior clinical-skills examination score, practice volume, and training-case breadth (+2.49 points per SD).

Conclusions:

By linking scores generated during required curricular practice to a platform-independent, assessor-blinded OSCE, this observational study provides criterion-related evidence for interpreting AMTES scores. The association was observed across OSCE-case strata, including a presenting complaint not represented among assigned AMTES cases, and remained positive when the score contribution from the complaint-matched training case was omitted. However, these between-student findings do not establish within-student transfer or a causal effect of practice, and residual confounding by prior ability remains possible. AMTES analytics warrant evaluation as a scalable, low-stakes formative signal to support faculty review and targeted educational support, but should complement rather than replace direct performance assessment. Prospective validation across cohorts, platform versions, and case sets is required before operational alerts or higher-stakes uses are considered.