JMIR Publications

ABSTRACT

Background:

Improving rigor and transparency measures should lead to improvements in reproducibility across the scientific literature, but assessing measures of transparency tends to be very difficult if performed manually.

Objective:

This study addresses an enhancement of the Rigor and Transparency Index (RTI v.2.0), which attempts to automatically assess the rigor and transparency of journals, institutions, and countries using manuscripts scored on criteria found in reproducibility guidelines (e.g., the MDAR checklist criteria).

Methods:

The RTI v.2.0 tracks 27 entity types using natural language processing techniques such as biLSTM CRF models and regular expressions, which allowed us to assess over 2 million papers accessed through PubMed Central (PMC).

Results:

Between 1997 and 2020 (where data was readily available in our dataset), rigor and transparency measures showed general improvement (RTI: 2.29 to 4.13), suggesting that authors are taking the need for improved reporting seriously. Top scoring journals in 2020 were Journal of neurochemistry (6.23), British journal of pharmacology (6.07) and Nature neuroscience (5.93). We extracted the institution and country of origin from author affiliations to expand our analysis beyond journals. Of institutions publishing more than 1,000 papers in 2020 (in the PMC OA set), Capital Medical University (4.75), Yonsei University (4.58), and University of Copenhagen (4.53) were top performers in terms of RTI. In terms of country-level performance, we found that Ethiopia and Norway consistently topped the RTI charts each year of countries with 100 or more papers. Additionally, we tested our assumption that the RTI may serve as a reliable proxy for scientific replicability (i.e., high RTI represents papers containing sufficient information for replication efforts). Using work by the Cancer Reproducibility Project, we determined that replication papers scored much higher (RTI = 7.61 ± 0.78) than the original papers (RTI = 3.45 ± 1.06), which according to the project all required additional information from authors to begin replication efforts.

Conclusions:

These results align with our view that the RTI may serve as a reliable proxy for scientific replicability. Unfortunately, RTI measures for journals, institutions, and countries all currently fall short of the replicated paper average. If we take the RTI of these replication studies as a target for future manuscripts, more work will be needed to ensure the average manuscript contains sufficient information for replication attempts.