JMIR Publications

ABSTRACT

Background:

Information regarding opioid use disorder (OUD) status and severity is valuable for patient care. Clinical notes provide valuable information for detecting and characterizing problematic opioid use, necessitating development of natural language processing (NLP) tools, which in turn requires reliably labeled OUD-relevant text and understanding of documentation patterns.

Objective:

To inform automated NLP methods, we aimed to develop and evaluate an annotation schema for characterizing OUD and its severity, and to document patterns of OUD-relevant information within clinical notes of heterogeneous patient cohorts.

Methods:

We developed a 27-class annotation schema to characterize OUD severity based on criteria from the Diagnostic and Statistical Manual of Mental Disorders (DSM-5). Two annotators reviewed clinical notes from key encounters of 100 adult patients with varied evidence of OUD, including patients with and without chronic pain, with and without medication treatment for OUD, and a control group. Annotations were completed at the sentence level. Severity scores were calculated based on annotation of note text with 18 classes aligned with DSM-5 criteria for OUD severity. Positive predictive values (PPVs) for OUD severity were determined.

Results:

We annotated 1,436 sentences from 82 patients; notes of 18 patients (61% of whom were controls) contained no relevant information. Inter-annotator agreement was above 70% for 11 of 15 batches of reviewed notes. Severity scores for control group patients were all 0. Among non-control patients, the mean (standard deviation) severity score was 5.1 (3.2), indicating moderate OUD, and the PPV for detecting moderate or severe OUD was 0.71. Progress notes and notes from emergency department and outpatient settings contained the most, and greatest diversity of information. Substance misuse and psychiatric classes were most prevalent and highly correlated across note types with high co-occurrence across patients.

Conclusions:

Implementation of the annotation schema demonstrated strong potential for inferring OUD severity based on key information in a small set of clinical notes and highlighted where such information is documented. These advancements will facilitate NLP tool development to improve OUD prevention, diagnosis, and treatment.