Accepted for/Published in: JMIR Formative Research
Date Submitted: Dec 17, 2025
Open Peer Review Period: Dec 17, 2025 - Feb 11, 2026
Date Accepted: Mar 9, 2026
(closed for review but you can still tweet)
Warning: This is an author submission that is not peer-reviewed or edited. Preprints - unless they show as "accepted" - should not be relied on to guide clinical practice or health-related behavior and should not be reported in news media as established information.
Scaling multimodal agentic AI in medical education: a multi-site cross-sectional study of simulation effectiveness in primary care
ABSTRACT
Background:
Conversational artificial intelligence (AI) systems offer potential solutions to traditional constraints in medical consultation skills training, including high costs, scheduling difficulties, and varied standardisation.
Objective:
There is limited evidence evaluating medical professionals' perceptions of AI generated patient interactions across multiple fidelity dimensions and assess the educational value of conversational AI for consultation skills training.
Methods:
Cross-sectional evaluation study at a UK medical school involving 179 participants (medical students and general practitioners) at 70 sites. Participants completed standardised clinical scenarios using SimFlow.ai conversational AI system, followed by structured questionnaires evaluating AI Realism, Medical Content, Educational Value, Feedback, and Usability domains. Data were analysed using descriptive statistics and non-parametric tests to assess domain performance and participant characteristics.
Results:
Medical Content received highest ratings (median 4.0, IQR 1.0 [4.0-5.0]), with 97.9% rating clinical plausibility highly. Educational Value was rated positively (IQR 1.0-2.0 [3.0-4.0] though AI Realism received moderate scores (median 3.0, IQR 2.0 [2.0-4.0]). Participants with prior AI experience gave significantly higher ratings for AI Realism than those without prior experience (mean 3.81 vs 3.07, p<0.05). Qualitative analysis revealed four themes: clinical authenticity, interactional limitations, educational potential, and implementation considerations.
Conclusions:
Conversational AI demonstrates strong capabilities in functional fidelity (clinical accuracy) despite limitations in conversational fidelity (realism). The technology shows promise for supplementary clinical skills training rather than higher-stakes assessment, with future development needed in dialogue naturalness and feedback capabilities. Clinical Trial: https://doi.org/10.17605/OSF.IO/NUGBD Not a clinical trial
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.