JMIR Publications

ABSTRACT

Background:

Large language models (LLMs) have shown remarkable capabilities in Natural Language Processing (NLP), especially in domains where labeled data is scarce or expensive, such as clinical domain. However, to unlock the clinical knowledge hidden in these LLMs, we need to design effective prompts that can guide them to perform specific clinical NLP tasks without any task-specific training data. This is known as in-context learning, which is an art and science that requires understanding the strengths and weaknesses of different LLMs and prompt engineering approaches.

Objective:

The objective of this study is to assess the effectiveness of various prompt engineering techniques, including two newly introduced types - heuristic and ensemble prompts, for zero-shot and few-shot clinical information extraction using pre-trained language models.

Methods:

This comprehensive experimental study evaluated different prompt types (simple prefix, simple cloze, chain of thought, anticipatory, heuristic, and ensemble) across five clinical NLP tasks: Clinical Sense Disambiguation, Biomedical Evidence Extraction, Coreference Resolution, Medication Status Extraction, and Medication Attribute Extraction. The performance of these prompts was assessed using three state-of-the-art language models: GPT-3.5, BARD(PALM-2), and LLAMA2. The study contrasted zero-shot with few-shot prompting and explored the effectiveness of ensemble approaches.

Results:

The study revealed that task-specific prompt tailoring is vital for high performance of LLMs for zero-shot clinical NLP. In Clinical Sense Disambiguation, GPT-3.5 achieved an accuracy of 0.96 with heuristic prompts and 0.94 in Biomedical Evidence Extraction. Heuristic prompts, alongside chain-of-thought prompts, were highly effective across tasks. Few-shot prompting improved performance in complex scenarios, and ensemble approaches capitalized on multiple prompt strengths. GPT-3.5 consistently outperformed BARD and LLAMA2 across tasks and prompt types.

Conclusions:

This study provides a rigorous evaluation of prompt engineering methodologies and introduces innovative techniques for clinical information extraction, demonstrating the potential of in-context learning in the clinical domain. These findings offer clear guidelines for future prompt-based clinical NLP research, facilitating engagement by non-NLP experts in clinical NLP advancements. To the best of our knowledge, this is one of the first works on the empirical evaluation of different prompt engineering approaches for clinical NLP in this era of generative AI, and we hope that it will inspire and inform future research in this area.