JMIR Publications

ABSTRACT

Background:

The recent availability of unstructured text narratives from law enforcement and coroner/medical examiners for nearly every violent death expands the potential for natural language processing (NLP) research into injury prevention.

Objective:

This paper examined the application of a compact large language model (LLM) to supervised text classification of coroner reports and police homicide reports on violent death to inform future applications of NLP to injury prevention.

Methods:

This analysis applied distilBERT, a compact LLM, to unstructured narrative data to simulate the impacts of pre-processing, volume and composition of training data on model performance, evaluated by F1-scores, precision, recall and the false negative rate. Model performance was evaluated for bias by race, ethnicity, and sex by comparing F1-scores across subgroups.

Results:

A minimum training set of 1,500 cases was necessary to achieve an F1-score of 0.6 and a false negative rate of .01-.05 with a compact LLM. Replacement of domain-specific jargon improved model performance while oversampling positive class cases to address class imbalance did not substantially improve F1 scores. Between racial and ethnic groups, F1-score disparities ranged from 0.2 to 0.25, and between male and female victims differences ranged from 0.12 to 0.2.

Conclusions:

Findings demonstrate that compact LLMs with sufficient training data can be applied to supervised NLP tasks to events with class imbalance in NVDRS unstructured police and coroner/medical examiner reports.Simulations of supervised text classification across the model-fitting process of pre-processing and training a compact LLM informed NLP applications to unstructured death narrative data.