JMIR Publications

ABSTRACT

Background:

Research has repeatedly shown that exposure to suicide-related news media content is associated with suicide rates, with some content characteristics likely having harmful and others potentially protective effects. Although good evidence exists for a few selected characteristics, systematic and large-scale investigations are missing. Moreover, the growing importance of social media, particularly among young adults, calls for studies on the effects of content posted on these platforms.

Objective:

This study applies natural language processing and machine learning methods to classify large quantities of social media data according to characteristics identified as potentially harmful or beneficial in media effects research on suicide and prevention.

Methods:

We manually labeled 3202 English tweets using a novel annotation scheme that classifies suicide-related tweets into 12 categories. Based on these categories, we trained a benchmark of machine learning models for a multi-class and a binary classification task. As models, we included a majority classifier, an approach based on word frequency (TF-IDF with a linear SVM) and two state-of-the-art deep learning models (BERT, XLNet). The first task classified postings into six main content categories, which are particularly relevant for suicide prevention based on previous evidence. These included personal stories of either (1) suicidal ideation and attempts or (2) coping and recovery, calls for action intending to spread either (3) problem awareness or (4) prevention-related information, (5) reporting of suicide cases, and (6) other tweets irrelevant to these five categories. The second classification task was binary, and separated postings in the 11 categories that refer to actual suicide, from postings in the off-topic category, which use suicide-related terms in another meaning or context.

Results:

In both tasks, the performance of the two deep learning models was very similar and better than the majority or the word frequency classifier. BERT and XLNet reached accuracy scores above 73% on average across the six main categories in the test set, and F1-scores in between 0.69 and 0.85 for all but the suicidal ideation and attempts category (F1 = 0.55). In the binary classification task, they correctly labeled around 88% of tweets as about suicide vs. off-topic, with BERT achieving F1-scores of 0.93 and 0.74, respectively. These classification performances were similar to human performance in most cases, and are comparable to the state-of-the-art on similar tasks.

Conclusions:

The achieved performance scores highlight machine learning as a useful tool for media effects research on suicide. The clear advantage of BERT and XLNet hints that there is crucial information about meaning in the context of words beyond mere word frequencies in tweets about suicide. By making data labeling more efficient, this work has enabled large-scale investigations on harmful and protective associations of social media content with suicide rates and help-seeking behavior.