JMIR Publications

ABSTRACT

Background:

Well-designed public health messages can help people make informed choices, while poorly designed messages, or messages designed to persuade, can confuse people, lead to poorly informed decisions, and diminish trust in health authorities and research. Communicating uncertainties to the public about the results of health research is challenging, and research is needed to evaluate effective ways to disseminate this important aspect of randomized trials.

Objective:

To evaluate people’s understanding of overall and statistical uncertainty when presented with alternative ways of expressing the results of a randomized trial.

Methods:

Two parallel, web-based, individually randomized trials (both 3×2 factorial designs) were conducted in U.S. and Norway. Participants were randomized to one of six versions of a text (summary) communicating results from a study that examined the effects of wearing glasses to prevent COVID-19 infection. The summaries differed in terms of how overall uncertainty ("GRADE language”, "plain language", or “no explicit language”) and statistical uncertainty (margin of error or not) were presented. Participants completed a web-based questionnaire exploring four co-primary outcomes: understanding overall and statistical certainty of evidence and harms, and sufficiency of evidence. Participants were non-glasses-wearing adults, recruited automatically from web-based research panels in the U.S. and Norway. The results of the trials were analyzed separately in a blinded manner and combined in meta-analysis.

Results:

In the U.S. and Norwegian trials, 730 and 497 individuals were randomized; data for 543 and 452 analyzed (i.e., after exclusions, mostly failing attention checks). More participants had a correct understanding of uncertainty when presented with plain language (US: 37/99 [37.4%]; Norway: 40/76 (52.6%]) than no explicit language (U.S: 18/86 [20.9%]; Norway: 34/80 [42.5%]). A similar positive effect was seen for GRADE language in U.S. (26/79 [32.9%]) but not in Norway (30/71 [42.3%]). There were only small differences between comparison groups for the outcome “understanding the certainty of important harms” that could have occurred by chance alone. Overall, plain compared to no explicit language also improved correct understanding of the sufficiency of the evidence (OR 2.05, 95% CI 1.17 to 3.57), but GRADE compared to no explicit language had no effect on this outcome (OR 1.36, 95% 0.80 to 2.31). Showing compared to not showing the margin of error improved understanding of statistical uncertainty (proportion who chose the correct answer) (Norway: 16/75 (21%) to 24/71 (34%) vs. 1/71 (1%) to 2/76 (3%); U.S.: 21/101 (21%) to 32/90 (36%) vs. 0/86 (0%) to 3/79 (4%)).

Conclusions:

Plain but not GRADE language was better than no explicit language in helping people understand overall certainty of evidence, though most participants did not correctly understand how sure they could be. Reporting margin of error improved interpretation of statistical uncertainty around the effect of glasses, but only for a minority of participants. Clinical Trial: ClinicalTrials.gov. NCT05642754