JMIR Publications

ABSTRACT

Background:

Clinical educators regularly use simulation-based education to help learners develop clinical reasoning, procedural skills, and communication strategies. However, differences between simulated and live clinical environments—such as missing case details, technical problems, or incomplete evaluation tools—can reduce realism and divert learners’ attention from the intended learning goals. While published design guidelines recommend pilot testing simulations, they do not explain how to systematically identify usability problems before piloting. Human factors evaluation methods can address this gap by identifying issues early and providing design insights.

Objective:

This tutorial describes our pragmatic usability testing protocol for evaluating educational simulations before piloting with learners. Using a telemedicine maternal care simulation as an example, we demonstrate how mixed-methods usability techniques can identify technical, workflow, and assessment-related issues in simulation design.

Methods:

We recruited obstetricians and family practitioners to test our prototype simulation. To measure multiple usability dimensions concurrently, we adapted several published instruments, including an Agile Task Analysis (ATA), the Single Ease Question (SEQ), the System Usability Scale (SUS), the Satisfaction with Simulation Experience Scale (SSES), and the NASA Task Load Index (NASA-TLX). We also developed a bespoke competency assessment rubric to measure tester performance in completing clinical telemedicine tasks. We recorded observations on the ATA while testers used a Think-Aloud protocol. We then debriefed testers and administered the post-test questionnaires. We summarized quantitative data using descriptive statistics and analyzed qualitative data using theory-based deductive coding and published usability heuristics.

Results:

Twelve testers identified 52 usability issues over two testing cycles; 15 issues required corrections before piloting with learners. Revisions included changes to standardised patient scripts, synthetic patient data, and scoring rubric instructions. Testers assigned a median SEQ score of six or higher to four of the six workflow steps; most struggled with unfamiliar telemedicine tasks or performing a virtual physical exam. The mean SUS score was 77.4, indicating above-average usability. The NASA-TLX subscale scores were highest for mental workload (mean 35.15) but appeared well calibrated for our target learners. The SSES scores were positive, suggesting faculty testers thought the simulation offered valuable learning opportunities. As a result of usability testing, we successfully piloted the simulation with 23 obstetrics and family medicine residents without encountering any implementation issues. Faculty easily scored residents on eight telehealth competencies. Most residents were entrustable with completing clinical tasks, apart from an unfamiliar virtual care checklist and assisting standardised patients with home blood pressure monitoring.

Conclusions:

Incorporating usability evaluation methods throughout the simulation development lifecycle can identify and mitigate design problems that might otherwise limit simulation fidelity and instructional effectiveness. Our tutorial showcases a modular mixed-methods approach that can be adapted to study an array of simulation types and research questions related to workflow, technology, and learning objectives. Clinical Trial: This was not a clinical trial. This simulation/tutorial was reviewed by the University of Oklahoma IRB # 13738