JMIR Publications

ABSTRACT

Background:

The credibility of online health information is a major challenge, where misleading or unsupported claims may influence health-related decisions.

Objective:

This study aimed to develop a credibility assessment method based on evidence-based medicine (EBM) for short health-related text samples derived from the Web. The method should be valid for any kind of textual content: either fragments derived from split documents or social media posts, or forum entries.

Methods:

We collected 664 Web documents related to COVID-19 and cardiology and split the documents into ‘paragraphs’ (text samples), each consisting of three consecutive sentences. Drawing inspiration from the literature, we constructed the initial annotation protocol and involved medical labelling experts with extensive experience in clinical and scientific practice to assess credibility of each sample. All samples were rated independently by two medical experts. We performed three iterations of protocol fine-tuning. Each iteration consisted of an assessment batch and the subsequent discussion by the expert panel. After each of the three iterations of the experiment, we assessed the final inter-rater agreement.

Results:

We assessed 8019 text samples across three iterations of the experiment and created an annotation protocol. The final protocol achieved inter-rater agreement of a Fleiss’ Kappa score of 0.45.

Conclusions:

We present the Medical Information Credibility (MedIC) method, the annotation protocol for labelling short health-related text samples derived from the Web. We propose protocol enhancements to avoid similar problems for future annotations. The method can be applied in a crowd-sourcing manner, thus making the process of data annotation more scalable. Our method together with the publicly available dataset is a good starting point for further research in online health disinformation detection.