JMIR Publications

ABSTRACT

Background:

Social media data is being increasingly utilized for population-level health research as it provides near real time access to large volumes of consumer generated data. In the recent past, a number of studies have explored the possibility of utilizing social media data, such as from Twitter, for monitoring prescription medication abuse. However, there is a paucity of annotated data or guidelines for data characterization discussing how information related to abuse-prone medications are presented in Twitter.

Objective:

Our primary objective in this paper is to discuss the creation of an annotated corpus suitable for training supervised classification algorithms for automatic classification of medication abuse-related chatter. We also describe the annotation strategies we used for improving inter-annotator agreement, a detailed annotation guideline and machine learning experiments illustrating the utility of the annotated corpus.

Methods:

We employed an iterative annotation strategy, involving inter-annotator discussions and the updating of an annotation guideline at each iteration to improve inter-annotator agreement for the manual annotation task. Using the grounded theory approach, we first characterized tweets into fine-grained categories and then grouped them into four broad classes—abuse/misuse, personal consumption, mention and unrelated. Following the completion of the manual annotations, we experimented with several machine learning algorithms to illustrate the utility of the corpus and also to generate baseline performance metrics for automatic classification on this data.

Results:

Our final annotated set consists of 16,433 tweets mentioning at least 20 abuse-prone medications including opioids, benzodiazepines, atypical antipsychotics, central nervous system stimulants and GABA analogues. Our final overall inter-annotator agreement was 0.86 (Cohen’s kappa), which represents high agreement. The manual annotation process revealed the variety of ways in which prescription medication misuse/abuse is discussed on Twitter, including expressions indicating co-ingestion, nonmedical use, non-standard route of intake, and consumption above prescribed dosages. Among machine learning classifiers, support vector machines obtained the highest automatic classification accuracy of 73.0% (95% CI: 71.4—74.5) over the test set (n=3,271).

Conclusions:

Our manual analysis and annotations of a large number of tweets have revealed types of information posted on Twitter about a set of abuse-prone prescription medications and their distributions. In the interests of reproducible and community-driven research, we have made our detailed annotation guidelines and the training data for the classification experiments publicly available, and the test data will be used in future shared tasks.