Currently accepted at: JMIR Medical Education
Date Submitted: Oct 20, 2025
Open Peer Review Period: Oct 21, 2025 - Dec 16, 2025
Date Accepted: Apr 7, 2026
(closed for review but you can still tweet)
This paper has been accepted and is currently in production.
It will appear shortly on 10.2196/86208
The final accepted version (not copyedited yet) is in this tab.
Warning: This is an author submission that is not peer-reviewed or edited. Preprints - unless they show as "accepted" - should not be relied on to guide clinical practice or health-related behavior and should not be reported in news media as established information.
Can AI Improve Medical School Admissions? A Reliability Analysis of AI-Generated Multiple Mini Interview (MMI) Stations.
ABSTRACT
Background:
Multiple Mini Interviews (MMIs) are widely used in medical school admissions to assess applicants’ non-academic attributes in a structured and reliable manner. However, the development of high-quality MMI stations is resource-intensive and dependent on expert input.
Objective:
This study explores the utility of artificial intelligence (AI) in the generation MMI stations for the Direct and Graduate Entry Medicine Program admissions process for domestic applicants at Monash Medical School.
Methods:
A total of 56 MMI stations from the 2025 admissions cycle were evaluated, including 17 AI-generated and 39 traditionally developed stations, administered across 824 domestic applicants for a total of 4,897 applicant-station interactions. We assessed station quality through both reliability (using Cronbach’s alpha to examine internal consistency) and discrimination capability (using standard deviation and range of scores) at the station level.
Results:
AI-generated stations exhibited slightly higher reliability (α = 0.8181) compared to existing stations (α = 0.8081), though this difference was not statistically significant. Both AI-generated and traditionally developed stations demonstrated variable discrimination capability, with some stations from each development method showing excellent combinations of high reliability and strong discriminatory power, while others exhibited ceiling effects that limited their discriminatory power. Importantly, a greater proportion of AI-generated stations achieved excellent reliability (α > 0.85), and a lower proportion demonstrated poor reliability (α < 0.75), compared to traditional stations, suggesting that AI-generated development may enhance the consistency and quality of MMI stations.
Conclusions:
Our findings presented here highlight the utility of AI as a useful tool for MMI station generation, offering a scalable approach that may reduce the resource burden on faculty while maintaining or enhancing psychometric quality for applicants. Ongoing quality assurance and evaluation remain essential to ensure fairness and validity across the admissions process.
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.