Accepted for/Published in: JMIR Formative Research
Date Submitted: May 19, 2026
Date Accepted: Sep 20, 2026
Artificial Intelligence for Debriefing Evaluation (AIDE): Examining the Use of Large Language Models for Structured Quantitative and Qualitative Analysis of Simulation Debriefings Using the DASH Framework — A Prospective Observational Study
ABSTRACT
Background:
Simulation-based education is a cornerstone of healthcare training, with debriefing representing the critical phase of the learning process. However, assessing debriefing quality remains challenging, relying largely on expert judgment and structured tools such as the Debriefing Assessment for Simulation in Healthcare (DASH). The emergence of artificial intelligence (AI), particularly Large Language Models (LLM), offers new opportunities for automated analysis of educational interactions.
Objective:
This study aimed to explore the feasibility and characteristics of AI-automated analysis of postsimulation debriefings using the DASH framework.
Methods:
We conducted a prospective, single-center, observational study at the Centre d’Apprentissage des Attitudes et Habiletés Cliniques (Université de Montréal). Postsimulation debriefings, conducted by a single instructor, involving anesthesia residents (postgraduate years 2–4) were audio-recorded, transcribed, and anonymized. Transcripts were analyzed using a natural language processing model (ChatGPT, OpenAI, GPT-5.3) with a structured prompt aligned with the six DASH dimensions. For each debriefing, the model generated a structured qualitative evaluation including a global rating on a 5-point ordinal scale, strengths, and areas for improvement supported by transcript excerpts. A cross-debriefing comparative analysis was also performed.
Results:
Thirty debriefings were included, with no exclusions or technical issues. The mean debriefing duration was 25.4 ± 4.8 minutes, and all audio recordings produced usable transcripts. All six DASH dimensions were consistently identified across sessions. For each debriefing, the model generated a structured qualitative evaluation including a global rating on a 5-point ordinal scale, identification of strengths and areas for improvement, and supporting transcript excerpts for each DASH dimension. Learning Climate and Maintaining Context were most frequently associated with high rating levels, whereas greater variability was observed in Debriefing Structure and Identification of Areas for Improvement. Cross-debriefing comparative analysis enabled identification of recurring patterns across sessions, including consistent learner engagement and variability in the formulation of actionable improvement objectives.
Conclusions:
This study demonstrates the feasibility of AI-assisted qualitative assessment of postsimulation debriefings using a LLM prompted according to the DASH framework. AI may represent a complementary tool to support pedagogical analysis, instructor reflection and faculty development in simulation-based education. While this exploratory study focused on textual transcript analysis without incorporating nonverbal communication or expert benchmarking, the findings support the feasibility of AI-assisted debriefing assessment and provide a foundation for future validation studies. Further research comparing AI-generated and expert evaluations could help refine these approaches and inform their integration into simulation-based education and faculty development.
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.