JMIR Publications

ABSTRACT

Background:

There is breast cancer-related medical information on social media, but there is no established method for objectively evaluating the quality of this information. PRHISM is a newly developed tool for objectively assessing the quality of health-related information on social media; however, there have been no reports evaluating its reliability and validity.

Objective:

The purpose of this study is to statistically examine the reliability and validity of PRHISM using videos about breast cancer treatment on YouTube.

Methods:

Sixty YouTube videos were selected on January 5, 2024, with the Japanese words for “breast cancer”, “treatment”, and “chemotherapy”, and assessed by six Japanese breast cancer experts. These evaluators independently evaluated the videos using PRHISM and an established tool for assessing the quality of health-related information, DISCERN, as well as through subjective assessments. We calculated inter-rater and intra-rater agreement among evaluators with confidence intervals, measuring agreement using weighted Cohen's kappa.

Results:

The inter-rater agreement for PRHISM overall quality was κ=0.52 (90% CI: 0.49, 0.55), indicating that the expected level of agreement, statistically defined by the lower limit of the 90% confidence interval exceeding 0.53, was not achieved. However, PRHISM demonstrated higher agreement compared to DISCERN overall quality, which had a κ=0.45 (90% CI: 0.41, 0.48). In terms of validity, the intra-rater agreement between PRHISM and subjective assessments by breast experts was κ=0.37 (95% CI: 0.14, 0.60), while DISCERN showed an agreement of κ=0.27 (95% CI: 0.07, 0.48), indicating fair agreement and no significant difference in validity.

Conclusions:

PRHISM has demonstrated sufficient reliability and validity for evaluating the quality of health-related information on YouTube, making it a promising new metric.