JMIR Publications

ABSTRACT

Objective:

Delirium in hospitalized patients is a syndrome of acute brain dysfunction. Diagnostic (ICD) codes are often used in studies using electronic health records (EHR), but are inaccurate. We sought to develop a more accurate method using Natural Language Processing (NLP) to detect delirium episodes based on unstructured clinical notes. Materials and Methods We collected 1.5M notes from >10K patients spanning 9 hospitals. Seven experts iteratively labeled 200,471 sentences. Using these, we trained three NLP classifiers: Support Vector Machine, Recurrent Neural Networks, and Transformer. Testing was performed on an external dataset. We also evaluated associations with delirium billing (ICD) codes, medications, orders for restraints and sitters, direct assessments (Confusion Assessment Method (CAM) scores), and in-hospital mortality. F1 scores, confusion matrices and AUC were used to compare NLP models. We used the Phi coefficient to measure associations with other delirium indicators.

Results:

The transformer NLP performed best: micro F1: 0.978, macro F1 0.918, positive AUC 0.984, negative AUC 0.992. NLP detections exhibited higher correlations (Phi) than ICD codes with deliriogenic medications (0.194], vs 0.073 for ICD codes); restraints and sitter orders (0.358 vs 0.177); mortality (0.216 vs 0.000); and CAM scores (0.256 vs -0.028). Discussion. Clinical notes are an attractive alternative to ICD codes for EHR delirium studies, but require automated methods. Our NLP model detects delirium with high accuracy, similar to manual chart review. Conclusion. Our NLP model can provide more accurate determination of delirium for large-scale EHR-based studies.