JMIR Publications

ABSTRACT

Background:

The implementation of Large Language Models (LLMs), such as BERT and GPT-4, has revolutionized the extraction of insights from unstructured text. These advancements have expanded into healthcare, analyzing social media for public health insights. Yet, drug discontinuation events (DDEs) detection remains underexplored. Identifying DDEs is crucial for understanding medication adherence and patient outcomes.

Objective:

The objective of this study is to provide a flexible framework for investigating various clinical research questions in data-sparse environments. We exemplify the utility of this framework by identifying DDEs and their root causes in an open-source online forum, medhelp.org, and by releasing the first open-source DDE datasets to aid further research in this domain.

Methods:

We employed several LLMs, including GPT-4 Turbo, GPT-4o, DeBERTa and BART among others, to detect and determine the root causes of DDEs in user comments posted on medhelp.org. Our study design included the use of zero-shot classification, which allows these models to make predictions without task-specific training. We split user comments into sentences and applied different classification strategies to assess the performance of these models in identifying DDEs and their root causes.

Results:

Among the selected models, GPT-4o performed the best at determining the root causes of DDEs, predicting only 12.9% of root causes incorrectly (Hamming Loss). Of the open-source models tested, BART performed the best at detecting DDEs by achieving an F1 score of 0.86, a false positive rate of 2.8%, and a false negative rate of 6.5% without any fine-tuning. The dataset comprised 10.7% DDEs, emphasizing the models’ robustness in an imbalanced data context.

Conclusions:

Our study demonstrates the effectiveness of open and closed-source LLMs, such as GPT-4o and BART, for detecting DDEs and their root causes from publicly accessible data through zero-shot classification. The robust and scalable framework we propose can aid researchers in addressing data-sparse clinical research questions. The release of open-access DDE datasets stands to stimulate further research and novel discoveries in this area.