JMIR Publications

ABSTRACT

Background:

Communicating uncertainties to the public about the results of health research is challenging.

Objective:

To test alternative ways to express overall and statistical uncertainty, using a study about glasses to reduce COVID risk as an example.

Methods:

Design Two online, parallel, individually randomized trials (both 3×2 factorial designs), randomized participants to different ways of presenting overall uncertainty ("GRADE language”, "plain language", or “no explicit language”) and statistical uncertainty (margin of error or none). Participants Non-glasses wearing adults, recruited from web-based research panels. Main outcome Understanding overall and statistical certainty of evidence. Secondary outcomes Perceived benefit, intended glasses-wearing behavior, assessment of provided information. Analysis Trials analyzed separately and combined in meta-analysis.

Results:

In the U.S. and Norwegian trials, 730 and 497 individuals were randomized; data for 543 and 452 analyzed (i.e., after exclusions, mostly failing attention checks): overall mean age 41 years (SD=15.4), 49% female, 48% with at least a college degree. Without margin of error, plain vs. no explicit language increased correct understanding of how sure one can be about the effect of wearing glasses: [US] 20.9% to 37.4% and [Norway] 42.5% to 52.6% (combined odds of correct understanding, 1.82, (95% CI 1.16 to 2.88)). Effect of GRADE vs. no explicit language was less certain: OR 1.29 (95% CI 0.80 to 2.06). Adding margin of error reduced understanding of plain (OR 0.51, 95% CI 0.27 to 0.99) and GRADE language (OR 0.82, 95% CI 0.42 to 0.1.60), with a shift from perceiving the evidence as “mixed but more unsure than sure” (correct) to “very unsure”. Adding margin of error increased the proportion correctly summarizing the glasses effect (i.e., "wearing glasses may reduce the chance of getting Covid a little, but might increase it a little") from 1%-3% correct to 21%-31% for the 3 kinds of language (overall OR 10.84, 95% CI 3.74 to 31.44). Plain vs. no explicit language increased the proportion correctly agreeing evidence was insufficient to be sure about the effect of glasses in the U.S. trial: 67.4% to 88.9%, risk difference 21.4%, 95% CI 9.8% to 33.1% but not in the Norwegian trial: 81.2% to 81.6 %, risk difference 0.3%, 95% CI -11.9% to 12.5%. The effect of GRADE vs. no explicit language was less certain (67.4% to 79.7% [US], and 81.2% to 79.5% [Norway]).

Conclusions:

Plain but not GRADE language was better than no explicit language in helping people understand overall certainty of evidence, though most participants did not correctly understand how sure they could be. Reporting margin of error reduced understanding of overall uncertainty by making people feel the evidence was even less certain. Reporting margin of error improved interpretation of statistical uncertainty around the effect of glasses, but only for a minority of participants. Clinical Trial: ClinicalTrials.gov. NCT05642754