JMIR Publications

ABSTRACT

Background:

The revised version of the risk of bias tool (RoB 2) overcomes some limitations compared with the original version, but concurrently introduces challenges in its application. Large language models (LLMs) may potentially assist the utilization of RoB 2. However, the exact methods and reliability remain uncertain

Objective:

This feasibility study aims investigate the capability of large language models in assessing the ROB of RCTs with RoB 2

Methods:

We systematically searched Cochrane reviews utilizing the RoB 2. Cochrane reviews were classified based on interested in adherence to intervention or assignment to intervention. From each category, 23 RCTs were randomly selected. The RoB 2 judgments reported the Cochrane reviews were used as the external validation standard. Three experienced reviewers were recruited to assess risk of bias of selected 46 RCTs using RoB 2. The reviewer judgments of six randomized controlled trials were selected to develop and optimize the prompt for the LLMs. The remaining 40 trials were used to establish the internal validation standard. Accuracy rate was calculated to reflect accuracy, both domain and signaling question; consistent assessment rate and Cohen’ κ were calculated to gauge consistency; and assessment time was calculated to measure efficiency.

Results:

Compared to Cochrane reviews, the LLMs' judgments demonstrated accuracy rates of 57.5% and 70% for Overall (assignment) and Overall (adhering), respectively. When compared to reviewer judgments, the LLMs' accuracy rates for Overall (assignment) and Overall (adhering) were 65% and 70.0%. The average accuracy rates for the remaining six domains were 65.2% (95% CI, 57.6%-72.7%) and 74.2% (95% CI, 64.7%-83.9%) when compared to Cochrane reviews and reviewers. The average accuracy rate in signaling level was 83.2% (95%CI: 77.5%-88.9%), consistent assessment rate is 85.2% (95% CI: 85.15%-88.79%). Compared to Reviewers, the LLMs conducted assessments 29.6 minutes (95% CI: 25.6-33.6) faster.

Conclusions:

LLMs were capable of rapidly assessing the risk of bias in RCTs using RoB 2, and exhibit a comparatively high level of accuracy. This suggests the potential utility of employing LLMs as adjunctive tools in the systematic review process.