JMIR Publications

ABSTRACT

Background:

Sentiment analysis of alternative tobacco products discussed on social media is a crucial area in tobacco control research. Large Language Models (LLMs) may hold the potential to streamline the time-consuming and labor-intensive process of human sentiment analysis.

Objective:

The accuracy of a language model in replicating human sentiment labeling of social media messages relevant to heated tobacco products (HTPs) was examined.

Methods:

ChatGPT was employed to classify 500 Facebook and 500 Twitter messages. Each set consisted of 200 human-labeled anti-HTPs, 200 pro-HTPs, and 100 neutral messages. The model evaluated messages up to 20 times to generate multiple response instances reporting its labeling decisions. The majority label from these responses was assigned as the model’s decision for each message. The model’s labeling decisions were then compared to those of human evaluators.

Results:

ChatGPT accurately replicated human sentiment labeling in 61.2% of Facebook messages and 57.0% of Twitter messages. Increasing the number of responses from the model improved the accuracy, with a single response yielding at least 83% accuracy with 20 responses. The model’s accuracy was higher for human-labeled anti-HTPs messages, compared to human-labeled pro-HTPs and neutral messages. Most of the misclassified human-labeled anti- and pro-HTPs messages were labeled by the ChatGPT as either neutral or irrelevant to HTPs.

Conclusions:

LLMs could be utilized to analyze sentiment in social media messages about HTPs. A potential challenge for using LLMs to analyze discourses related to HTPs on social media could be the underrepresentation of messages that express positive attitudes towards these products.