JMIR Publications

ABSTRACT

Background:

Identifying NID cases using International Classification of Diseases (ICD) billing codes is often imprecise, while manual chart reviews are labor-intensive. ML models can leverage unstructured electronic health records (EHR) to detect subtle NID indicators, process large data volumes efficiently, and reduce misclassification. While accurate NID classification is needed for research and clinical decision support, using unstructured notes for this purpose remains underexplored.

Objective:

To develop and validate a machine learning (ML) model to identify neuroinfectious diseases (NID) from unstructured patient notes.

Methods:

Clinical notes from patients who had undergone lumbar puncture were obtained using the EHR of an academic hospital network (Mass General Brigham, MGB), with half associated with NID-related diagnostic codes. Ground-truth was established by chart review with six NID-expert physicians. NID keywords were generated with regular expressions, and extracted texts were converted into bag-of-words representations using n-grams (n=1, 2, 3). Notes were randomly split into training (80%) and hold-out testing (20%) sets. Feature selection was performed using logistic regression with L1 regularization. An extreme gradient boosting (XGBoost) model classified NID cases, and performance was evaluated using the Area Under the Receiver Operating Curve (AUROC) and the Precision-Recall Curve (AUPRC). Performance of the natural language processing (NLP) model was contrasted with LLaMA 3.2 auto-regressive model on the MGB test set. The NLP model was additionally validated on external data from an independent hospital (Beth Israel Deaconess Medical Center, BIDMC).

Results:

This study included 3,000 patient notes from MGB from January 22, 2010, to September 21, 2023. Of 1,284 initial n-gram features, 342 were selected, with the most significant features being ‘meningitis,’ ‘ventriculitis,’ and ‘meningoencephalitis.’ The XGBoost model achieved an AUROC of 0.98 (95% CI: 0.96 - 0.99) and AUPRC of 0.89 (95% CI: 0.83 - 0.94) on MGB test data. In comparison, NID identification using ICD-billing codes showed high sensitivity (0.97) but poor specificity (0.59), overestimating NID cases. LLaMA 3.2 improved specificity (0.94) but had low sensitivity (0.64) and an AUROC of 0.80. In contrast, our NLP model balanced specificity (0.96) and sensitivity (0.84), outperforming both methods in accuracy and reliability on MGB data. When tested on external data from BIDMC, the NLP model maintained an AUROC of 0.98 (95% CI: 0.96 - 0.99), with an AUPRC of 0.78 (95% CI: 0.66 - 0.89).

Conclusions:

The NLP model accurately identifies NID cases from clinical notes. Validated across two independent hospital datasets, the model demonstrates feasibility for large-scale NID research and cohort generation. With further external validation, our results could be more generalizable to other institutions.