JMIR Publications

ABSTRACT

Background:

Generative large language models (LLMs) have the potential to revolutionize medical education by generating tailored learning materials, enhancing teaching efficiency, and improving learner engagement. However, the application of LLMs in healthcare settings, particularly for augmenting small datasets in text classification tasks, remains underexplored, particularly for cost- and privacy-conscious applications that do not permit the use of third-party services such as OpenAI’s ChatGPT.

Objective:

This paper explores the use of open-source LLMs, such as Large Language Model Meta AI (LLaMA) and Alpaca models, for data augmentation in a specific text classification task related to hospital staff surveys.

Methods:

The surveys were designed to elicit narratives of everyday adaptation by frontline radiology staff during the initial phase of the COVID-19 pandemic. The study evaluates the effectiveness of various LLMs, temperature settings, and downstream classifiers in improving classifier performance.

Results:

The overall best-performing combination of LLM, temperature, classifier, and number of augments is LLaMA 7B at temperature 0.7 using Robustly Optimized BERT Pretraining Approach (RoBERTa) with 100 augments, with an average the Area Under the Receiver Operating Characteristic curve (AUC) of [0.87] ±[0.02: 1 standard deviation]. The results demonstrate that open-source LLMs can enhance text classifiers' performance for small datasets in healthcare contexts, providing promising pathways for improving medical education processes and patient care practices.

Conclusions:

The study demonstrates the value of data augmentation with open-source LLMs, highlights the importance of privacy and ethical considerations when using LLMs, and suggests future directions for research in this field.