Accepted for/Published in: JMIR mHealth and uHealth
Date Submitted: Aug 17, 2023
Date Accepted: Mar 27, 2024
ChatGPT versus Human Researchers—Efficacy in Identifying Relevant Studies on m-health Interventions for Improving Medication Adherence in Ischemic Stroke Patients during Systematic Reviews: A Comparative Analysis
ABSTRACT
Background:
ChatGPT emerged as a potential tool for researchers, aiding in various aspects of research. One such application was the identification of relevant studies in systematic reviews. However, a comprehensive comparison of the efficacy of relevant study identification between human researchers and ChatGPT has yet to be determined.
Objective:
To compare the efficacy of ChatGPT and human researchers in identifying relevant studies on medication adherence improvement using m-health interventions in ischemic stroke patients during systematic reviews.
Methods:
The Preferred Reporting Items for Systematic Reviews and Meta-Analyses were used as a guideline for this study. Four electronic databases, including CINAHL Plus with Full Text, Web of Science, and PubMed/Medline, were searched to identify articles published from inception until 2023 using search terms based on Medical Subject Headings (MeSH) generated by human researchers versus ChatGPT. The authors independently screened the titles, abstracts, and full text of the studies identified through separate searches conducted by human researchers and ChatGPT. The comparison encompassed several aspects, including the ability to retrieve relevant studies, accuracy, efficiency, limitations, and challenges associated with each method.
Results:
Six articles based on search terms generated by human researchers were included in the final analysis. While, ten studies were included based on search terms generated by ChatGPT, with 60% (n = 6) of them overlapping. The precision in accurately identifying relevant studies was higher in human researchers (0.86) than in ChatGPT (0.77). However, when considering the time required for both humans and ChatGPT to identify relevant studies, ChatGPT significantly outperformed human researchers as it took less time to identify relevant studies.
Conclusions:
Our comparative analysis highlighted the strengths and limitations of both approaches. Ultimately, the choice between human researchers and ChatGPT depended on the specific requirements and objectives of each review, but the collaborative synergy of both approaches held the potential to advance evidence-based research and decision-making in the healthcare field. Clinical Trial: Not applicable.
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.