Maintenance Notice

Due to necessary scheduled maintenance, the JMIR Publications website will be unavailable from Wednesday, July 01, 2020 at 8:00 PM to 10:00 PM EST. We apologize in advance for any inconvenience this may cause you.

Who will be affected?

Accepted for/Published in: JMIR Cardio

Date Submitted: Nov 9, 2025
Date Accepted: Sep 24, 2026

The final, peer-reviewed published version of this preprint can be found here:

Comparison of Rule-Based Natural Language Processing and Diagnosis Code Approaches to Detecting Bleeding in the Electronic Health Record: Algorithm Validation Within a Retrospective Cohort Study

Draper EW, Anderson JA, Shah S, Brown L, Hanson MR, Negley KD, Lutz JM, Anderson AE, O'Meara JG, Jakowenko ND, McBane RD

Comparison of Rule-Based Natural Language Processing and Diagnosis Code Approaches to Detecting Bleeding in the Electronic Health Record: Algorithm Validation Within a Retrospective Cohort Study

JMIR Cardio 2026;10:e87424

DOI: 10.2196/87424

PMID: 43062208

Comparison of Rule-Based Natural Language Processing and Diagnosis Code Approaches to Detecting Bleeding in the Electronic Health Record: Algorithm Validation within a Retrospective Cohort Study

  • Evan W Draper; 
  • Jan A Anderson; 
  • Surbhi Shah; 
  • Leilia Brown; 
  • Margaret R Hanson; 
  • Kristin D Negley; 
  • Jennifer M Lutz; 
  • Andrea E Anderson; 
  • John G O'Meara; 
  • Nicholas D Jakowenko; 
  • Robert D McBane

ABSTRACT

Background:

Anticoagulant use is prevalent and increasing. High quality anticoagulation management balances individual thrombotic and bleeding risks to optimize care. An automated algorithm to accurately extract major bleeding events from the electronic health record (EHR) of a large health-care system would be extremely useful for tracking outcomes of anticoagulated patients.

Objective:

Assess the accuracy of a Natural Language Processing (NLP) approach compared against, and in conjunction with a diagnosis code-based strategy for detecting major bleeding from the EHR.

Methods:

Patients with at least one prescription for apixaban, dabigatran, edoxaban, rivaroxaban, or warfarin (1/1/2019 - 12/31/2023) were analyzed using a natural language processing (NLP) algorithm and diagnosis codes to detect major bleeding. Separate approaches and algorithms were created for Central Nervous System (CNS) Bleeding, Critical Site Bleeding, and bleeding with a hemoglobin drop or transfusions (Other Major Bleeding). The NLP, diagnosis codes, and combined NLP with diagnosis codes algorithms were compared to a gold standard, manual chart review. Bleeding events were adjudicated by at least two reviewers.

Results:

The final analysis identified 111,338 patients prescribed outpatient anticoagulation over the study timeframe. Six hundred and one patients of 111,338 were manually adjudicated. The NLP algorithm performed similar to diagnosis code approaches for central nervous system bleeding, f1-scores 0.84 to 0.97, and Other Major Bleeding, f1-scores 0.69 to 0.77. The NLP algorithm outperformed one of the diagnosis code approaches for Critical Site Bleeding, f1-score 0.83 vs 0.27. The combined NLP and diagnosis code approach had excellent specificity of 0.99 (0.96-1.00) but only 0.68 (0.57-0.78) sensitivity while the combined approach considering either algorithm had specificity of 0.93 (0.87-0.96) and 0.96 (0.88-0.99) sensitivity.

Conclusions:

Electronic tools can accurately screen the EHR for major bleeding outcomes from large patient cohorts within a multi-site healthcare network.


 Citation

Please cite as:

Draper EW, Anderson JA, Shah S, Brown L, Hanson MR, Negley KD, Lutz JM, Anderson AE, O'Meara JG, Jakowenko ND, McBane RD

Comparison of Rule-Based Natural Language Processing and Diagnosis Code Approaches to Detecting Bleeding in the Electronic Health Record: Algorithm Validation Within a Retrospective Cohort Study

JMIR Cardio 2026;10:e87424

DOI: 10.2196/87424

PMID: 43062208

Download PDF


Request queued. Please wait while the file is being generated. It may take some time.

© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.