JMIR Publications

ABSTRACT

Background:

Sentiment analysis, in general, and in Cantonese, in particular, remain a significant yet challenging task in natural language processing. One major barrier is Cantonese’s lack of standardised corpus and its nature as a spoken language.

Objective:

Our study investigated using ChatGPT for sentiment analysis in online Cantonese counseling text and compared its performance with other mainstream methods.

Methods:

This study investigated the application of ChatGPT/GPT for Cantonese sentiment analysis using transcripts from an online text-based counseling service in Hong Kong. A total of 131 individual counseling sessions, with 6,169 messages between counselors and help-seekers were included in this study. First, a codebook was developed for human annotation. Then a simple prompt, “Is the sentiment of this Cantonese text positive, neutral, or negative? Respond with the sentiment label only.”, was given to ChatGPT 3.5 and GPT-4 to label each message’s sentiment. ChatGPT’s performance was compared with one lexicon-based method and three state-of-the-art models.

Results:

The accuracy in identifying positive, neutral, and negative feelings of ChatGPT 3.5 and GPT-4 was 92.1% and 95.3%, respectively, which outperformed the lexicon-based methods and machine learning models.

Conclusions:

ChatGPT/GPT stands out among many existing text analysis techniques in terms of accuracy and could be considered a useful tool for analyzing Cantonese sentiments.