JMIR Publications

ABSTRACT

Background:

The Medical College Admission Test (MCAT) has been central to medical school admissions in North America, though its necessity in holistic selection processes remains debated. The COVID-19 pandemic’s suspension of MCAT testing allowed institutions to explore alternative admissions criteria. Additionally, global challenges in test administration underscore the vulnerability of systems dependent on a single standardized test.

Objective:

This study aimed to (1) assess whether including MCAT scores in admissions decisions improves prediction of medical school performance compared to GPA and interview-based selection, and (2) evaluate whether machine learning (ML) can generate viable MCAT score predictions when testing is unavailable.

Methods:

We conducted a retrospective cohort study of 1,898 applicants to the Lebanese American University School of Medicine (2009–2023). Among 349 admitted students with MCAT scores, we compared admission composites including versus excluding MCAT in relation to academic outcomes (Med 1–4 final grades) and clinical performance (Med 1–2 OSCE scores). Students were stratified into tertiles to examine tier-specific effects. To model an MCAT alternative, we trained an ensemble ML model (LASSO, Kernel Ridge, Gradient Boosting, Elastic Net, LightGBM) using data from 1,583 applicants (2009–2020) to predict MCAT scores from cumulative GPA, core GPA, interview scores, merit points, and honors participation. The model was validated on 315 applicants admitted during MCAT suspension (2021–2023), and its impact on admission rankings and performance correlations was evaluated.

Results:

Including MCAT in the admissions composite did not meaningfully improve prediction of overall academic performance (r=0.63 with MCAT vs. r=0.62 without; P=.634). However, it significantly weakened prediction of clinical skills (OSCE: r=0.37 with MCAT vs. r=0.48 without; P<.001). In tertile analyses, MCAT modestly improved prediction among top-performing students but eliminated predictive validity in the bottom tertile (r=0.12, p=.206 vs. r=0.28 without MCAT; P=.003). The ML model explained 36% of MCAT variance (R²=0.36; RMSE≈10% of score range). When predicted MCAT scores were incorporated into admissions during the pandemic cohorts, correlations with subsequent performance declined across all outcomes. Although rankings based on predicted MCAT strongly correlated with original rankings (r=0.8), 6.3%–9.4% of admission decisions would have changed.

Conclusions:

Within a structured holistic admissions process emphasizing GPA and interviews, MCAT scores added minimal incremental validity for academic performance and reduced prediction of clinical skills, particularly among lower-tier applicants where selection decisions are most consequential. ML-based MCAT predictions introduced sufficient error to degrade admissions decisions and are not suitable substitutes in high-stakes contexts. Medical schools with robust holistic frameworks may sustain student quality during MCAT disruptions without relying on predictive modeling.