JMIR Publications

ABSTRACT

Background:

While automation is recognised for saving time in conventional systematic reviews, a trust gap and steep learning curve persist regarding its accuracy and ease of use. It remains unknown if these adoption barriers apply to the more complex process of systematic grey literature reviews, especially given the rapid evolution of large language models.

Objective:

This study seeks to identify the preferred features of systematic grey literature reviews (SGLRs) generated using an automation tool (automated SGLRs).

Methods:

Participants aged 18 years or older with experience using evidence from systematic reviews were recruited. A cross-sectional online Best-Worst Scaling questionnaire was distributed through the authors’ contacts. Descriptive, multinomial, and mixed logit analyses were used to estimate respondents' preferred features of automated SGLRs.

Results:

A total of 168 respondents, comprising researchers, academicians, healthcare professionals, health policymakers, students, and postgraduates, completed the study. Multinomial logit analysis found that the top three preferred features for automated SGLRs were i) high precision and sensitivity comparable to manual methods, ii) acknowledgement and detailed explanation on the AI algorithms, and iii) involvement of ≥ 2 independent human reviewers in the process. The mixed logit model revealed significant heterogeneity in respondents' preferences for automated SGLR features. Further subgroup analysis indicated that respondents aged 18-29 and 40-49 considered detailed explanations and acceptance by everyone to be more important features of automated SGLRs. In contrast, academicians significantly preferred the detailed explanations feature of automated SGLRs.

Conclusions:

The study concludes that automation tool developers should ensure the accuracy of automated SGLRs, ensure algorithmic transparency, and integrate human-in-the-loop processes to build user trust and drive adoption of automated SGLRs.