Accepted for/Published in: Journal of Medical Internet Research
Date Submitted: Jan 14, 2026
Date Accepted: Sep 24, 2026
Natural Language Processing to Detect Risky Alcohol Use in Electronic Health Records: Improving Model Performance Using Data from the Annotation Process
ABSTRACT
Background:
High-risk alcohol use and alcohol use disorder (AUD) are significant contributors to surgical risk and complications yet remain under-identified in surgical populations. While electronic health records (EHRs) contain relevant information about patients' alcohol use in clinical notes, manual chart review is time-intensive and impractical at scale. Natural language processing (NLP) offers a path to automated identification, but alcohol-related risk classes are highly imbalanced in clinical populations, making multi-class classification particularly challenging.
Objective:
To develop an automated NLP pipeline for four-class alcohol risk classification using clinical notes in a surgical population, addressing class imbalance by leveraging intermediate annotation evidence generated during the human labeling process.
Methods:
We manually labeled 1,000 surgical patients drawn from a large academic health system into four ordinal alcohol risk classes: No-Alcohol, Low-Risk, High-Risk, and Probable-Dependence, using a structured annotation process with six trained annotators and expert-developed guidelines. We developed a multi-stage NLP pipeline operating on three years of clinical notes per patient. First, a rules-based preprocessing stage filtered notes to clinically relevant sections containing alcohol-related content. Second, a BioClinicalBERT-based multilabel classifier ("auto-annotator") was fine-tuned to reproduce the 26-category evidence annotations created by human labelers at the section level, producing a structured 26-dimensional probability vector for each section. Finally, a bidirectional long short-term memory (LSTM) network classified patients into the four risk categories by processing each patient's ordered sequence of auto-annotation vectors, allowing the model to capture temporally accumulating evidence across clinical encounters.
Results:
Human annotators achieved 93.7% first-pass agreement overall (Fleiss’ kappa 0.8891), but with agreement as low as 67.5% for Probable-Dependence cases. The four-class LSTM classifier achieved a weighted-average F1 of 0.87 and a macro-averaged F1 of 0.77 on a held-out test set of 195 patients. One-vs-Rest AUC scores were 0.97 for No-Alcohol, 0.90 for Low-Risk, 0.92 for Probable-Dependence, and 0.70 for High-Risk. Collapsed to a binary high/low risk classification, the model achieved an accuracy of 0.92 and macro F1 of 0.80. Performance was comparable across biological sex without explicit sex-specific training.
Conclusions:
We used a two-stage NLP pipeline to identify four ordinal classes of alcohol use risk in surgical patients. Four-class alcohol-use classification presents a highly-imbalanced problem that constrains the availability of high-importance but rare labels in the dataset. We addressed this data scarcity by using intermediate annotation evidence as well as final labels to enrich the feature space for the classification task. The pipeline was able to separate patient records into four strata of alcohol-use risk, with “high-risk” patients being most likely to be misclassified into an adjacent class.
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.