JMIR Publications

ABSTRACT

Background:

Multicenter electronic health records (EHR) can support quality improvement and comparative effectiveness research in critical care. However, limitations of EHR-based research include challenges in abstracting key clinical variables, including a patient’s level of consciousness.

Objective:

The objective of our study was to develop a natural language processing (NLP) model to predict the Glasgow Coma Scale (GCS) scores from daily EHR notes.

Methods:

The study included adult patients (≥18 years) admitted to Massachusetts General Brigham (MGB) hospitals (2017-2024) and patients from the MIMIC-III database (Medical Information Mart for Intensive Care-MIMIC III 2001-2012) v1.4. A dataset of all patients from both institutions was split into train/hold-out test (70%/30%) sets. Variables consisted of daily notes, age, sex, admission type, and an anesthetic/sedative drug indicator. We trained a pooled ordinal regression model (ordinalNet) with an elastic net penalty to predict the lowest daily level of consciousness across three classes: severe (GCS 3-8), moderate (GCS 9-12) and mild (GCS 13-15), and a pooled linear model to predict continuous GCS scores (3–15). Gold standard GCS was obtained from structured flowsheet data. External generalizability was assessed using a single-institution ordinal model trained on MGB and tested on MIMIC. For all models, a two-stage approach was used in which the minimum GCS extracted from notes was applied when available; otherwise, the model’s prediction was used. Following post-hoc calibration, ordinal and linear model performance was evaluated on the hold-out test sets using the areas under the receiver characteristic curve (AUROC) and precision-recall curve (AUPRC); and root mean square error (RMSE) and Pearson correlation, respectively.

Results:

Our modeling cohort included 144,813 patients (MGB = 122,992; MIMIC = 21,821) with 1,432,074 days of hospitalization, between training and testing sets; average age 63 [SD 18] years and balanced sex distribution. The pooled ordinalNet achieved AUROC and AUPRC [95% CI] of 0.94 [0.94-0.95] and 0.70 [0.69-0.71], with two-stage performance of 0.89 and 0.67. The single-institution ordinal model achieved AUROC 0.83 [0.83-0.83] and AUPRC 0.68 [0.68-0.69], with two-stage performance of 0.81 and 0.67. The pooled linear model achieved RMSE 2.26 [2.26-2.26] and correlation 0.76 [0.76-0.76], and the two-stage linear model achieved RMSE 2.35 and correlation 0.73. Unresponsiveness was the most important predictor of lower GCS, followed by propofol administration, mechanical ventilation or intubation, and failure to follow commands. Features associated with higher GCS included patient alertness, responsiveness, and orientation.

Conclusions:

Pooled ordinal and linear models accurately predict GCS across institutions, and two-stage models maintain strong performance when documentation is incomplete. Our NLP-based models can enable large-scale phenotyping of neurological assessments and critical care research studies.