JMIR Publications

ABSTRACT

Background:

While most current medication error prevention systems are rule-based, these systems may result in alert fatigue due to poor accuracy. Previously, we developed a machine learning (ML) model based on Taiwan’s National database to address this issue. However, the international transferability of such a model is unclear.

Objective:

To validate the international transferability of a machine learning model for medication error prevention using U.S. hospital data

Methods:

The study cohort included 667,572 outpatient prescriptions from two large U.S. academic medical centers. Our ML model was applied to build the Original model (O model), the Local model (L model), and the Hybrid model (H model). The O model was built using the data of 1.34 billion outpatient prescriptions from the Taiwan’s National database. A validation set with 60,000 prescriptions (9%) was first randomly sampled, and the remaining 607,572 prescriptions (91%) served as the local training set for the L model. The H model utilized the association values with a higher frequency of co-occurrence among the O model and L model. A testing set with 600 prescriptions was classified into ‘substantiated’ and ‘unsubstantiated’ by two independent physician reviewers, and then used to assess model performance.

Results:

The inter-rater agreement was significant in terms of classifying prescriptions as ‘substantiated’ and ‘unsubstantiated’ (kappa, 0.91; 95 percent confidence interval, 0.88 to 0.95). With thresholds from 0.5 to 1.5, the alert accuracy ranged from 75% to 78% for the O model, 76% to 78% for the L model, and 79% to 85% for the H model.

Conclusions:

Our ML model has good international transferability among U.S. hospital data. Further augmentation with local hospital data could improve the model’s accuracy.