Accepted for/Published in: Journal of Medical Internet Research
Date Submitted: Aug 3, 2026
Date Accepted: Oct 6, 2026
A Fine-Tuned Large Language Model Workflow to Support Risk of Bias Assessment: Development, External Validation, and Reviewer Verification
ABSTRACT
Background:
Given that randomized controlled trials (RCTs) are the gold standard of effectiveness research, assessing their risk of bias (RoB) becomes an indispensable step in systematic reviews. Yet this task remains time‑consuming and prone to reviewer subjectivity. While large language models (LLMs) may accelerate the process, they lack transparent, traceable judgments linked to trial reports.
Objective:
To bridge methodological gaps and improve both efficiency and consistency, this study developed and externally validated TRoB‑1, a fine‑tuned open‑weight LLM workflow for RoB assessment of RCTs.
Methods:
TRoB‑1 was developed specifically for the Cochrane risk of bias tool version 1 (RoB 1), which is the most widely adopted instrument for critical appraisal of RCTs. The workflow was built on two-stage supervised fine-tuning of Gemma 2-9B-Instruct using 4-bit quantized low-rank adaptation. The training set comprised 4321 RoB records extracted from systematic reviews of chronic obstructive pulmonary disease, diabetes, and hypertension. The validation set consisted of 161 records from recently published RCTs covering a broader disease area. The reference standard was established by two independent reviewers. A clinician blinded to the reference standard verified TRoB-1-assisted assessments through a locally hosted web interface. Using the same reference standard, the performance of TRoB-1 raw outputs was compared with that of the Gemma 2 base workflow, and reviewer-verified outputs were compared with raw TRoB-1 outputs.
Results:
TRoB-1 successfully generated structured outputs for all 161 validation records. Raw TRoB-1 judgments achieved an accuracy of 73.9% (95% CI 66.5%-80.7%), a macro-F1 score of 0.603 (95% CI 0.513-0.686), and an MCC of 0.461 (95% CI 0.337-0.581). By comparison, the Gemma‑2 base workflow scored 60.2%, 0.518, and 0.324 on the same three metrics, respectively. The pairwise difference in accuracy remained significant (P<.001). After reviewer verification, accuracy increased to 82.0% (95% CI 75.8%-87.6%), macro-F1 to 0.709 (95% CI 0.616-0.784), MCC to 0.615 (95% CI 0.494-0.726), and Gwet’s AC1 to 0.765 (95% CI 0.677-0.840). Reviewer verification corrected 13 initially incorrect judgments without changing any correct judgment to incorrect (P<.001). The F1 score for judgments of high-risk of bias increased from 0.600 to 0.784.
Conclusions:
By integrating task‑specific fine‑tuning, structured outputs, and reviewer verification, the TRoB‑1 workflow enhances efficiency and transparency while maintaining performance. The TRoB‑1 web interface provides a practical platform for integrating RoB into evidence synthesis, particularly for living systematic reviews where timely updates are critical. Clinical Trial: Not applicable
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.