JMIR Publications

ABSTRACT

Background:

The Military Healthcare System mandates medical documentation at all echelons of care, however care providers in combat situations must prioritize lifesaving measures over record-keeping, leading to information gaps. Effective human machine teaming (HMT) solutions designed to autonomously document care delivery will serve as future force multipliers in tactical combat casualty care (TCCC) environments.

Objective:

To address this challenge, the United States Army Institute of Surgical Research commenced an effort to prototype HMT systems designed to passively document care delivery in TCCC environments. However, common artificial intelligence (AI) performance evaluation methods do not represent the temporal, repetitive and context dependent nature of real-world TCCC delivery. It was essential to conduct comprehensive assessments of the AI functions in a timely, synchronized manner. During the initial prototyping phase, developers were provided with annotated datasets from seventy-five TCCC simulations and given six months to develop their algorithms.

Methods:

To assess performance, the research team leveraged a reserved dataset. In the first phase of the assessment a standardized, repeatable performance methodology and framework was leveraged to evaluate individual algorithms that detect: (1) injury location; (2) visible medical objects, and (3) treatments administered by the care provider. Detection effectiveness included four metrics: modified accuracy; precision; recall; and F1 scores. Algorithm processing efficiency was evaluated by calculating lag time scores. A final composite score was used to quantify performance differences among the algorithms within a specific detection category. The second phase of the evaluation integrated multiple algorithms into a centralized orchestration framework; enabling synchronized execution and consolidated outputs. System resource usage and throughput metrics were evaluated to characterize the efficiencies. Quantified memory consumption and processing unit resource utilization were assessed, followed by benchmarking the edge compute orchestration framework.

Results:

Results to date revealed that Medical Object Detection achieved the highest performance (mean F1≈0.42, range 0–0.71). Injury Detection and Localization showed lower performance (mean F1≈0.27, range 0–0.60), with higher recall than precision. Medical Procedure Detection yielded procedure-level mean F1 scores from 0.00 to 0.31 and simulation-level means from 0.00 to 0.33. Stronger results were observed for Nasopharyngeal Airway and Chest Seal Application (mean F1≈0.28, 0.31) medical procedures. The results to date are preliminary and serve as illustrative examples of the evaluation framework outputs.

Conclusions:

The preliminary results highlight the evaluation framework’s end-to-end, standardized results across core algorithm functions. While algorithm performance to date is modest, the framework demonstrates its capacity to capture both variability and recurring patterns across simulations, thereby highlighting strengths, limitations, and areas requiring refinement. It enables reproducible, cross data set comparisons, allowing evaluators to quantify algorithm performance. By leveraging both simulation-level evaluations and detection specific performance aggregated across simulations, the framework enables targeted identification of underperforming areas, supporting iterative and strategic AI model refinement.