JMIR Publications

ABSTRACT

In a proof-of-concept study, we sought to assess whether ChatGPT could accurately perform a biomedical labeling tasks that had been previously performed by 5 human annotators. ChatGPT was given annotation instructions to find adverse events (AEs) that were identical to those given to the human annotators and was asked to analyze the same dataset of 10,000 posts to a forum discussing the use of a cannabis product called delta-8-THC. ChatGPT replicated the human annotations with a high accuracy for both general adverse event reports (97.7% agreement; Kappa = 0.95) and instances of serious adverse event reports e.g., a life-threatening event or hospitalization (98.0% agreement; Kappa = 0.95). Given the accuracy, we extended ChatGPT’s capabilities to annotate the entire archive of relevant data (N=76,247 posts), resulting in the discovery of substantially more potential adverse events and providing insights into the trends and patterns of AE reports over an extended period that would have required substantial effort for human annotators. The results hold promise for the use of ChatGPT for biomedical text analysis in a manner that extends the capabilities of human annotators and offers advantages over traditional natural language text classification techniques.