JMIR Publications

ABSTRACT

Background:

Competency-based medical education (CBME) increasingly relies on data-driven assessment to support early identification of at-risk students, yet few studies have applied machine learning (ML) alongside traditional inferential statistics to the same longitudinal medical education dataset to compare their predictive and explanatory value.

Objective:

This study aimed to examine medical students' academic performance using both explanatory (inferential) and predictive (ML) approaches, to identify academic determinants of competency attainment, clarify their sequence across the curriculum, and to develop a data-driven model supporting academic monitoring and timely intervention.

Methods:

This retrospective study included medical students enrolled at the Lebanese American University Gilbert and Rose-Marie Chaghoury School of Medicine (LAU-SOM) from 2013 to 2025. Predictors included demographic characteristics, premedical admission metrics (grade point average, Medical College Admission Test subscores, interview score), and longitudinal knowledge and clinical-skills outcomes across four medical school years, along with competency gap-matrix classifications. Sequential multivariable linear, logistic, and multinomial regression models tested statistical relationships among variables. In parallel, nine tree-based ML algorithms were trained and compared using admission-only versus sequential predictor sets, with model selection via cross-validated R2 (regression) or weighted F1 score (classification), and interpretation via SHapley Additive exPlanations (SHAP) values.

Results:

The cohort was 50.7% male. Inferential models showed that early performance (Med 1) was predicted mainly by premedical GPA and MCAT sub-scores, while performance in later years was increasingly predicted by prior in-program academic outcomes, particularly Med 2 IFOM and OSCE scores. ML models mirrored this pattern: admission-only models showed moderate accuracy for Med 1–2 outcomes (test R2 up to 0.603) but weak performance for Med 3–4 outcomes (test R2 as low as 0.064). Incorporating sequential data improved prediction, raising test R2 to 0.328–0.831 across Med 2–4 outcomes, with the largest gain for Med 3 knowledge performance (R2 from 0.152 to 0.672). Sequential predictors similarly improved binary gap-status classification, increasing weighted F1 score from 0.632 to 0.773 and accuracy from 0.629 to 0.776.

Conclusions:

Integrating longitudinal, sequentially updated academic performance data markedly improves both statistical explanation and machine-learning prediction of student outcomes and competency gaps. This dual explanatory-predictive framework offers a practical, data-driven early-warning system that can be embedded into digital learning-management platforms to support real-time academic monitoring and timely intervention within CBME