JMIR Publications

ABSTRACT

Background:

Many junior doctors are unprepared to manage acutely ill patients in the emergency department. The setting is often stressful, and urgent treatment decisions are needed. Overlooking symptoms and making wrong choices may lead to significant patient morbidity or death, and it is essential to ensure that junior doctors are competent. Virtual Reality (VR) software can provide standardized and unbiased assessment, but solid validity evidence is necessary before implementation.

Objective:

To gather validity evidence for using 360-degree VR videos with integrated multiple-choice questions (MCQ) to assess emergency medicine skills.

Methods:

Five full-scale emergency medicine scenarios were recorded with a 360-degree video camera, and MCQs were integrated into the scenarios played in a head-mounted display. We invited three groups of medical students with different experience levels to participate: first-year medical students (novice group), last-year medical students without emergency medicine training (intermediate group), and last-year medical students with completed emergency medicine training (experienced group). Each participant's total test score was calculated based on the number of correct MCQ answers (maximum score of 28), and the groups’ mean scores were compared. The participants rated their experienced presence in emergency scenarios using the Igroup Presence Questionnaire (IPQ) and the cognitive workload with the NASA Task Load Index (NASA-TLX).

Results:

We included 61 medical students from December 2020 to December 2021. The experienced group had significantly higher mean scores than the intermediate group (23 vs. 20, P=0.038), and the intermediate group had significantly higher scores than the novice group (20 vs. 14, P<0.001). The contrasting groups’ standard-setting method established a pass/fail score of 19 points (68% of the maximum score possible). Inter-scenario reliability was high, with a Cronbach’s alpha of 0.82. The participants experienced the VR scenarios with a high degree of presence with an IPQ score of 5.83 (scale 1-7) and the task to be mentally demanding with a NASA-TLX score of 13.30 (scale 1-21).

Conclusions:

This study provides validity evidence to support using 360-degree VR scenarios to assess emergency medicine skills. The students evaluated the VR experience as mentally demanding with a high degree of presence, suggesting that VR is a promising new technology for emergency medicine skills assessment.