Accepted for/Published in: Journal of Medical Internet Research
Date Submitted: Oct 4, 2021
Date Accepted: Jun 13, 2022
Warning: This is an author submission that is not peer-reviewed or edited. Preprints - unless they show as "accepted" - should not be relied on to guide clinical practice or health-related behavior and should not be reported in news media as established information.
Wisdom of the Experts versus Wisdom of the Crowd in Hospital Quality Ratings: Cross-Sectional Analyses of Google Ratings
ABSTRACT
Background:
Popular online portals provide free and convenient access to user-generated quality reviews. Centers for Medicare and Medicaid Services (CMS) also provide patients with Hospital Compare Star Ratings (HCSR), a single public measure of hospital quality aggregating multiple quality dimensions. Consumers often use crowdsourced hospital ratings on platforms such as Google to select hospitals, but it is unknown if these ratings reflect a comprehensive measure of clinical quality.
Objective:
We analyze if Google online quality ratings, which reflect the wisdom of the crowd, are associated with HCSR, which reflect the wisdom of the experts. CMS revised the methodology of assigning star ratings to hospitals. Therefore, we analyze these associations before and after the 2021 revisions of the CMS rating system.
Methods:
We extracted Google ratings using Application Programming Interface (API) in June 2020. The HCSR data of April 2020 (before the revision of HCSR methodology) and April 2021 (after the revision of HCSR methodology) were obtained from CMS’ Hospital Compare (HC) website. We also extracted scores for the individual components of hospital quality for each of the hospitals in our sample using the code provided by HC. Fractional Response Model (FRM) was used to estimate the association between Google Ratings and HCSR and individual components of quality.
Results:
Results indicate that Google ratings are statistically associated with HCSR (P<.001) after controlling for hospital level effects. A one star improvement in CMS ratings before the change in methodology (after the change in methodology) is expected to increase the Google ratings by 0.145 (0.135) on average (95% CI 0.127- 0.163; P<.001, 95% CI 0.116-0.153; P<.001). The analyses with individual components of hospital quality reveal that Google ratings are not associated with components of HCSR that require medical expertise such as ‘Safety of care’ or ‘Readmissions’. The revised CMS rating system ameliorates previous partial inconsistencies in association between Google ratings and component scores of HCSR.
Conclusions:
Overall, crowd sourced Google hospital ratings are informative about expert CMS hospital quality ratings and several individual quality components that are easier for patients to evaluate. Therefore, hospitals should not expect improvements in quality metrics that require expertise to assess such as safety of care and readmission to result in improved Google star ratings. Hospitals can benefit from using crowd-sourced ratings as timely, easily available, and dynamic indicators of their quality performance.
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.