JMIR Publications

ABSTRACT

Background:

Bleeding adverse drug events (ADEs), particularly among older adult patients on antithrombotic therapy, are a significant concern in hospital settings. These events often go undetected using traditional rule-based methods relying on structured data from electronic medical records, underscoring the need for more effective detection approaches.

Objective:

This study aimed to develop and evaluate a natural language processing (NLP) model to accurately detect and categorise bleeding events in older adult inpatients’ discharge summaries. Specifically, it would identify ADEs related to antithrombotic therapy and compare the NLP model’s performance with Boolean algorithms based on International Classification of Diseases, 10th Revision (ICD-10) codes.

Methods:

Clinicians manually annotated 400 discharge summaries, comprising 65,706 sentences, into four categories: ‘no bleeding’, ‘clinically significant bleeding’, ‘severe bleeding’ and ‘history of bleeding’. These annotations were used to train and validate two detection models: an NLP model using binary logistic regression and support vector machine classifiers and a rule-based model using ICD-10 codes specific to bleeding ADEs. We assessed both models’ performance using accuracy, precision, recall, F1 score and the area under the curve (AUC) from receiver operating characteristic (ROC) analysis. Manual annotations served as the gold standard.

Results:

The NLP model outperformed the rule-based model, especially in identifying ‘clinically significant’ and ‘severe bleeding’. The NLP model achieved macro-averages of 0.81 for accuracy and 0.80 for the F1 score. It also demonstrated high precision in distinguishing current bleeding ADEs from past ones, with a strong true positive rate and minimal false positives.

Conclusions:

This study highlights a significant advance in using artificial intelligence for healthcare, with the NLP model surpassing traditional ICD-10 coding for detecting bleeding ADEs in electronic medical records. The NLP model provides a more precise tool for clinical decision-making involving older adult patients on antithrombotic therapy.