JMIR Publications

ABSTRACT

Background:

Fatal drug overdose surveillance informs prevention but is often delayed due to autopsy report processing and death certificate coding. Autopsy reports contain narrative text describing scene evidence and medical history (similar to preliminary death scene investigation reports) and may serve as early data sources for identifying fatal drug overdoses. To facilitate more timely fatal overdose reporting, natural language processing (NLP) was applied to narrative text from autopsies.

Objective:

This study aimed to develop an NLP-based model predicting the likelihood that an autopsy report narrative describes an accidental or undetermined fatal drug overdose.

Methods:

Autopsies for all manners of death (2019-2021) were obtained from the Tennessee Office of the State Chief Medical Examiner. Text was extracted from autopsy reports (in portable document format files) using optical character recognition. Three common narrative text sections were identified, concatenated, and preprocessed (bag-of-words) with term frequency-inverse document frequency scoring. Logistic regression, support vector machine (SVM), random forest, and gradient boosted trees classifiers were developed and validated. Autopsies from 2019-2020 were used for training (95%) and calibration (5%), and 2021 for testing. Model discrimination was evaluated using area under the receiver operating characteristic (AUROC), precision, recall, F1 score, and F2 score (prioritizes recall over precision). Calibration was performed using logistic regression (Platt scaling) and evaluated using the Spiegelhalter z-test. Shapley Additive exPlanations (SHAP) values were generated for models compatible with the method. In a post-hoc subgroup analysis of the random forest classifier, model discrimination was evaluated by forensic center, race, and age at death.

Results:

A total of 17,342 autopsies (34% cases, 66% controls) were used for model development and validation. The training set included 10,215 autopsies (33% cases, 67% controls), calibration set had 538 autopsies (34% cases, 66% controls), and test set had 6,589 autopsies (37% cases, 63% controls). The vocabulary set contained 4,002 terms. All models showed excellent performance (AUROC ≥0.95, precision ≥0.94, recall ≥0.92, F1 ≥0.94, and F2 ≥0.92). The SVM and random forest classifiers achieved the highest F2 scores (SVM F2=0.948; random forest F2=0.947). The logistic regression and random forest were calibrated (P=.95 and P=.85 respectively), while the SVM and gradient boosted trees classifiers were miscalibrated (P=.029 and P<.001 respectively). “Fentanyl” and “accident” had the highest SHAP values. Post-hoc subgroup analyses revealed lower F2 scores for autopsy reports from forensic centers D and E. Lower F2 scores were also observed for the American Indian, Asian, ≤14, and ≥65 subgroups, but larger sample sizes are needed to validate these findings.

Conclusions:

The random forest classifier may be suitable for identifying potential accidental and undetermined fatal overdose autopsies. Operationalizing this classifier could enable the early detection of accidental and undetermined fatal drug overdoses.