JMIR Publications

ABSTRACT

Background:

Trauma registries are crucial for evaluating the prognosis of injuries and quality of trauma care. However, the calculation of Injury Severity Score (ISS) remains a predominantly manual process, which is labor-intensive, potentially error-prone, and difficult to scale. Automating this process is a critical step toward enhancing the efficiency and quality of trauma registry data.

Objective:

This study conducts a feasibility test on the use of large language models (LLMs) to automatically calculate ISS from free-text discharge diagnoses. We specifically investigate the efficacy of retrieval-augmented generation (RAG) to enhance LLM accuracy by incorporating knowledge from the Abbreviated Injury Scale (AIS). We compare the performance of baseline LLMs against models augmented with both a standard RAG and a novel, domain-specific RAG framework.

Methods:

In this retrospective validation study, we utilized de-identified, free-text discharge diagnoses from trauma patients at Chang Gung Memorial Hospital (Linkou, Taiwan) in 2022. Against a physician-verified gold standard, we compared the performance of three systems: (1) baseline LLMs, (2) LLMs with standard RAG, and (3) LLMs with the domain-specific RAG, for which the retriever was fine-tuned on trauma-related ICD-10 codes. Performance was evaluated on a physician-verified test set across exact ISS prediction, tolerance-based accuracy, and ISS severity categories.

Results:

Baseline state-of-the-art LLMs achieve modest exact-match (EM) accuracy, ranging from 50.0% to 58.0%. Performance improved with standard RAG integration (from 52.0% to 61.3%) and increased substantially with the domain-specific RAG (from 62.0% to 73.0%). Performance gains were most pronounced in patients with moderate to severe injuries. GPT-4.1 with domain-specific RAG reached the highest EM accuracy (73.0%). Tolerance-based and categorical analyses confirm consistent substantial gains with the RAG frameworks in clinically relevant ranges.

Conclusions:

Large language models augmented with a domain-specific retriever can accurately and automatically calculate the Injury Severity Score from free-text discharge diagnoses. This technology provides a scalable solution to alleviate the administrative burden of manual coding, enhance standardization, and improve the quality of trauma registry data.