JMIR Publications

ABSTRACT

Background:

Best Practice Alerts (BPAs) are alert messages to physicians in the Electronic Health Record (EHR) that are used to encourage appropriate utilization of healthcare resources. While these alerts are helpful in both improving care and reducing costs, the alerts cause significant burden on physicians. The development of Large Language Models provides an opportunity to selectively identify target patients and reduce physician alert fatigue.

Objective:

We propose a new application for Large Language Models (LLMs) in Quality Improvement to selectively identify patients for Best Practice Alerts. An LLM system can increase alert efficiency as well as reduce physician alert fatigue. In this paper we present an example case of an LLM used to optimize patient selection for a BPA encouraging prescription of deep venous thrombosis (DVT) prophylaxis.

Methods:

Using a simulated population from the MIMIC III dataset we present an LLM system that selectively targets patients appropriate for a DVT prophylaxis BPA by identifying and excluding patients experiencing acute bleeding. The LLM system uses GPT-3 to create a synthetic training set and fine-tune a classification BioMed-RoBERTa model. Model classification results were compared to a physician reviewer.

Results:

The model achieved impressive accuracy, favoring high sensitivity or specificity depending on the characteristics of the training dataset. An ROC AUC of 0.89 with a sensitivity of 88% and specificity of 77% was achieved when negation examples were not included in the synthetic training set. In contrast the model achieved an ROC AUC of 0.85 with a sensitivity of 56% and specificity of 90% when negation examples were included. The models increased BPA efficiency by 34% and 22% respectively compared to a simple rule-based algorithm.

Conclusions:

These results demonstrate how language models can significantly improve efficiency of Best Practice Alerts over current rule-based algorithms, minimizing physician alert fatigue. We provide an example application that uses a simple RoBERTa model deployed with minimal compute power. Larger models (ex. GPT3, PaLM, etc) could exhibit even superior performance. We anticipate that the development of HIPAA-compliant platforms for larger language models will even further expand the potential of LLMs to revolutionize the field of Quality Improvement.