Accepted for/Published in: JMIR Formative Research
Date Submitted: Feb 15, 2024
Date Accepted: Sep 20, 2024
Public Health Discussions on Social Media: An Evaluation of Automated Sentiment Analysis Methods
ABSTRACT
Background:
Sentiment analysis is the most popular artificial intelligence method to mine and examine text. Popular sentiment analysis tools were used to analyze YouTube comments from videos discussing the opioid epidemic, inform health policy, and improve population health.
Objective:
This study compared manually coded sentiment scores to scores from three automated sentiment analysis tools: VADER, TEXT2DATA, and the tone summary measure from LIWC-22.
Methods:
Evaluation methods included descriptive statistics, ROC curves, confusion matrices, Cohen’s kappa, accuracy, specificity, precision, sensitivity (recall), and F1 harmonic means.
Results:
Overall, LIWC’s tone sentiment analysis measure most closely matched manual coding (88% F1 score), followed by VADER (83% F1 score) and TEXT2DATA (82% F1 score).
Conclusions:
We recommend applying these tools to social media analysis, considering imbalances in positive and negative affect, comment length, and other practical considerations, such as costs and programming needs.
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.