Comparison of Rule-Based Natural Language Processing and Diagnosis Code Approaches to Detecting Bleeding in the Electronic Health Record: Algorithm Validation within a Retrospective Cohort Study
ABSTRACT
Background:
Anticoagulant use is prevalent and increasing. High quality anticoagulation management balances individual thrombotic and bleeding risks to optimize care. An automated algorithm to accurately extract major bleeding events from the electronic health record (EHR) of a large health-care system would be extremely useful for tracking outcomes of anticoagulated patients.
Objective:
Assess the accuracy of a Natural Language Processing (NLP) approach compared against, and in conjunction with a diagnosis code-based strategy for detecting major bleeding from the EHR.
Methods:
Patients with at least one prescription for apixaban, dabigatran, edoxaban, rivaroxaban, or warfarin (1/1/2019 - 12/31/2023) were analyzed using a natural language processing (NLP) algorithm and diagnosis codes to detect major bleeding. Separate approaches and algorithms were created for Central Nervous System (CNS) Bleeding, Critical Site Bleeding, and bleeding with a hemoglobin drop or transfusions (Other Major Bleeding). The NLP, diagnosis codes, and combined NLP with diagnosis codes algorithms were compared to a gold standard, manual chart review. Bleeding events were adjudicated by at least two reviewers.
Results:
The final analysis identified 111,338 patients prescribed outpatient anticoagulation over the study timeframe. Six hundred and one patients of 111,338 were manually adjudicated. The NLP algorithm performed similar to diagnosis code approaches for central nervous system bleeding, f1-scores 0.84 to 0.97, and Other Major Bleeding, f1-scores 0.69 to 0.77. The NLP algorithm outperformed one of the diagnosis code approaches for Critical Site Bleeding, f1-score 0.83 vs 0.27. The combined NLP and diagnosis code approach had excellent specificity of 0.99 (0.96-1.00) but only 0.68 (0.57-0.78) sensitivity while the combined approach considering either algorithm had specificity of 0.93 (0.87-0.96) and 0.96 (0.88-0.99) sensitivity.
Conclusions:
Electronic tools can accurately screen the EHR for major bleeding outcomes from large patient cohorts within a multi-site healthcare network.
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.