JMIR Publications

ABSTRACT

Background:

Findings from many randomized controlled trials (RCTs) in cardiovascular surgery have not been reproduced or are small, statistically fragile, and potentially-influenced by funding source. They also demonstrate variable endpoint selection and protocol adherence, all of which may influence reproducibility of their reported results.

Objective:

To assess the reproducibility of RCTs conducted in the most commonly-studied disease states in cardiovascular surgery that compared at least one open surgical procedure to an alternative open, interventional catheter-based, or sham procedure or medical therapy.

Methods:

We will conduct a meta-epidemiological study of the reproducibility of RCTs using umbrella review methodology. We designed the study protocol using the Preferred Reporting Items for Systematic review and Meta-Analysis-Protocols statement and published recommendations on conducting umbrella reviews. We will search MEDLINE, EMBASE, and the Cochrane Central Register of Systematic Reviews from their inception for systemic reviews published in the English-language that included data from phase-three RCTs (if defined in this manner by authors) or those with a clinically-important primary outcome (instead of a surrogate or feasibility outcome) and compared an open cardiovascular surgery (performed in the most commonly-studied disease states across these specialities, including coronary artery disease, valvular or electrical heart disease, aortic root pathologies/dissections, carotid artery stenosis, abdominal aortic aneurysms, and peripheral artery disease) to either an alternative open, catheter-based, or sham procedure or medical therapy. Two investigators will independently review the individual RCTs from these systematic reviews and extract data and assess risk of bias (using the Cochrane Risk of Bias-2 tool), the fragility or reverse fragility index (which provides a measurement of the statistical robustness of the trials), and the authors’ interpretation of the results (also referred to as spin). RCTs will be classified as the original study if it was the first chronological study (based on participant enrollment date) to evaluate the compared interventions and as reproduction attempts if they re-evaluate the compared interventions in a different set of participants. The interpretation of a study’s results will be based on the findings of its primary endpoint and be used to assess reproducibility. We will also create multivariable regression models to determine whether the adjusted odds of reproducibility is independently predicted by 12 different variables.

Results:

We registered the study protocol, but have not yet conducted the search. We aim to complete the study by the end of 2026 and submit for peer-review in early 2027.

Conclusions:

Study results will help guide which cardiovascular surgery procedures should be considered for adoption or de-adoption and which are in need of reproduction studies. They will also identify characteristics of an original RCT in this field that predicts future study reproducibility before reproducibility studies are published. Clinical Trial: The protocol for our study was submitted for registration on PROSPERO on February 7, 2026.