JMIR Publications

ABSTRACT

Background:

Understanding the intricate factors influencing patient well-being necessitates a focused examination of addiction status within the broader context of health determinants. The successful identification of patients' addiction profiles equips clinical care teams to address addiction-related issues more effectively, enabling targeted support and ultimately improving patient outcomes.

Objective:

This study investigates the application of the generative pre-trained transformer (GPT) model for extracting tobacco, alcohol, and substance addiction information from patient discharge summaries in zero-shot and few-shot learning settings. This study contributes to the evolving landscape of healthcare informatics by showcasing the potential of advanced language models in extracting nuanced information critical for enhancing patient care

Methods:

The main data source for analysis in this paper is Medical Information Mart for Intensive Care III (MIMIC-III) dataset. Among all notes in this dataset, we focused on discharge summaries. Prompt engineering was undertaken, involving an iterative exploration of diverse prompts. Leveraging carefully curated examples and refined prompts, we investigate the model's proficiency through zero-shot as well as few-shot learning setting.

Results:

The presented results highlight the contrasting performance of GPT in extracting mentions of tobacco, alcohol, and substance use in both zero-shot and few-shot learning scenarios. In the zero-shot setting, the accuracy for extraction of tobacco, alcohol, and substance use mentions is notably high. However, in the few-shot setting, the accuracy diminishes significantly. On the contrary, few-shot learning led to significant increase in devising the status of addiction compared to zero-shot learning with significant increase in recall and F1-score. However, this improvement comes at the cost of a reduction in precision in both addiction mention and status extraction.

Conclusions:

Excellence of zero-shot learning in precisely extracting addiction mentions demonstrates its effectiveness in situations where comprehensive recall is paramount. Conversely, few-shot learning offers advantages when accurately determining the status of addiction is the primary focus, even if it involves a trade-off in precision.