JMIR Publications

ABSTRACT

Background:

In recent years, the advent of large language models (LLMs) such as GPT-4 has opened new avenues in natural language processing (NLP), particularly in understanding and generating human-like text. These models have shown promise in various domains, including healthcare, where precise and empathetic communication is crucial.

Objective:

This aims to evaluate the performance of GPT-4, a generative large language model (LLM), against a discriminative model that requires retraining to identify patient intent within a cancer patient contact center, emphasizing the influence of in-context learning on understanding complex patient queries.

Methods:

We used a dataset of 430,355 sentences from telephone consultations with cancer patients between 2016 and 2020. The data were divided into training (300,000 sentences up to 2019) and testing (1,000 randomly selected sentences from 2020 onward) sets. The performance of GPT-4 was evaluated using both zero-shot and few-shot approaches and compared with discriminative models, such as LSTM (long short-term memory) and BERT (bidirectional encoder representations from transformers).

Results:

GPT-4, which uses only a few examples (a few shots), attained a remarkable accuracy of 85.2 %, considerably outperforming the LSTM and BERT models, which achieved accuracies of 73.7% and 71.3%, respectively. This finding highlights the potential of LLMs with in-context learning capabilities to better understand the complex intentions of oncology patients. These findings emphasize the potential of LLMs, particularly GPT-4, for interpreting complicated patient interactions during cancer-related telephone consultations. The limitations in achieving high classification accuracy highlight the significance of accurate category delineation and prompt optimization to maximize the potential of LLMs in healthcare communication.

Conclusions:

This study presents GPT-4 as a potential tool for enhancing the effectiveness and consistency of patient interactions during telephone oncological consultations. The ongoing development of advanced LLMs offers exciting opportunities to improve patient understanding and refine informational support for cancer patients.