JMIR Publications

ABSTRACT

Background:

Social media is today seriously emerging as a source of information on post-marketing pharmacovigilance (PV), i.e. adverse events occurring with marketed drugs. Social media grew in importance in the very last years but still remains largely unexploited nowadays. Quite a few researchers published their work on either drug names or adverse effects (AE) recognition or drug adverse reactions (ADR) information extraction. However, a Gold Standard, consisting in manual annotations of the ADR by human experts from the corpus extracted from social media, was not systematically implemented and its quality is not always assessed on its own. When existing, the sample size is arbitrary only and doesn’t rely on statistical arguments. This questions the statistical reliability of the results. At last, some of the extraction methods just take into account any co-occurrence of a drug entity with one or several AE entitie(s) which leads to extract general information such as the indication of the drug in addition to regular ADR.

Objective:

In our work, we propose a standardized protocol for the evaluation of a software extracting information purely on ADRs.

Methods:

Messages from French health forums are extracted (evaluation dataset). Drug and AE entities recognition are based on lexicons: Racine Pharma thesaurus and MedDRA terminology respectively. NLP-based techniques automate the ADR information extraction (relation between the Drug and AE entities). Several concepts of ADRs will be chosen. The corpus for evaluation is a random sample of the messages containing these concepts from the evaluation dataset. Two persons experienced in medical terminology will manually annotate the corpus (Gold Standard). The study outcomes will be the precision and recall and their confidence intervals at 95%. For each concept, these outcomes will be computed against the Gold Standard. Necessary and sufficient sample size will be calculated to ensure statistical confidence in the assessed results. Gold Standard in its-self will be evaluated through Kappa inter-annotators agreements. Further analyses will enable to explore the granularity in the terminologies.

Results:

The automated ADR information extraction in the corpus for evaluation is already finished; the ADRs’ concepts as well as the sample for the Gold Standard are selected. The Gold Standard is completed, we currently proceed to the data analyses and the study results are expected in 2018.

Conclusions:

This protocol is the first one that used standardized statistical methods to create the corpus for evaluation of a NLP tool, thus ensuring necessary statistical power of the assessed results. Once completed, our ADR information extraction software should interest different actors of our society (Health agencies, pharmaceutical companies, general public).