JMIR Publications

ABSTRACT

Background:

Mass-casualty and operational triage require rapid decisions under uncertainty, yet real-world labeled decision data from these settings are scarce, heterogeneous, and difficult to collect at scale. Immersive virtual reality (VR) simulation provides a controlled way to generate structured human triage data in realistic high-consequence scenarios.

Objective:

To evaluate whether simulation-derived human triage examples improve the performance and safety profile of a lightweight open-weight large language model (LLM) beyond zero-shot prompting and explicit triage guidance alone, and whether the pattern is observed in simulation datasets organized around Sort, Assess, Life-Saving Interventions, Treatment/Transport (SALT) and Tactical Combat Casualty Care (TCCC).

Methods:

We performed a retrospective computational evaluation using structured cases derived from immersive VR triage simulations. Four prompting conditions were evaluated: zero-shot, context/guideline only, simulation-derived few-shot examples only, and few-shot examples plus context. In the TCCC-oriented analysis, few-shot examples were evaluated from both a matched TCCC-derived source and a cross-source SALT-derived source mapped to the four TCCC labels. Outcomes included accuracy; F1 score, which balances precision and recall, summarized as macro-F1 and weighted-F1; per-category F1; under-triage; critical under-triage; pooled failure modes; attribute-stratified performance; few-shot sensitivity; and prompt-elicited model confidence. TCCC row-level evaluations contain repeated presentations and are interpreted descriptively.

Results:

For SALT, test accuracy was 68.2% with zero-shot prompting, 68.4% with context alone, 71.1% with few-shot examples, and 79.1% with few-shot plus context; macro-F1 was 51.7%, 57.2%, 70.7%, and 83.8%, respectively. In TCCC, few-shot-plus-context accuracy was 87.0% with matched-source examples and 88.1% with cross-source examples; the corresponding context-only accuracies were 68.6% and 70.9%. The gain from adding examples beyond context alone was 10.7 percentage points for SALT, 18.5 points for matched-source TCCC, and 17.2 points for cross-source TCCC (pooled descriptive gain, 17.8 points). Category-level results showed large gains for EXPECTANT with simulation examples, while SALT DELAYED remained the weakest category. SALT critical under-triage decreased from 11.1% with few-shot prompting to 0.4% with few-shot plus context. TCCC under-triage decreased from 9.8% to 5.5% in the matched-source analysis and from 9.8% to 4.9% in the cross-source analysis; critical under-triage was 0% in both few-shot-plus-context source conditions. Prompt-elicited confidence remained high even when predictions were wrong and did not provide a reliable case-level safety filter.

Conclusions:

Simulation-derived human examples improved lightweight LLM triage performance beyond explicit framework guidance alone. The effect was observed with both matched-source TCCC examples and cross-source SALT-derived examples, and the strongest conditions reduced safety-relevant under-triage. These findings support immersive simulation as a practical source of reusable human decision data that preserve observed variation and experiential judgments not necessarily encoded in written guidance or reproduced by model-generated synthetic cases, with potential relevance to other high-consequence domains where real-world decision data are scarce. TCCC estimates are descriptive because many rows are repeated presentations of a much smaller number of patient identities, and the results do not establish readiness for autonomous or real-time clinical triage.