JMIR Publications

ABSTRACT

Background:

Artificial Intelligence (AI) is becoming increasingly popular in the scientific field, as it allows to analyze extensive datasets, summarize results, and assist writing academic papers.

Objective:

This study investigates the role of AI in Systematic Literature Review (SLR) writing, focusing on its contributions and limitations in each step of its development.

Methods:

Four AI platforms were tested on their ability to reproduce four PRISMA-based SLRs. We used Connected Papers and Elicit to perform research of relevant articles; then we assessed Elicit and ChatPDF's ability to extract and organize information contained in the articles’ text. Finally, we tested Jenni AI’s capacity to compose a SLR.

Results:

Neither Connected Papers nor Elicit provided the totality of the results found using the PRISMA method. Data extracted from Elicit resulted accurate in 51.40±31.45% of cases and imprecise in 13.69±17.98%; 22.37±27.54% of responses were missing, while 12.51±14.70% were incorrect. Data extracted from ChatPDF were accurate in 60.33± 30.72% of cases and imprecise in 7.41±13.88%; 17.56±20.02% of responses were missing, and 14.70±17.72% were incorrect. Jenni AI’s generated content exhibited satisfactory language fluency and technical proficiency, while resulting insufficient in defining methods, elaborating results, and stating conclusions.

Conclusions:

The PRISMA method continues to exhibit clear superiority in terms of reproducibility and accuracy at each step of conducting a SLR.5 While AI can save time and assist with repetitive tasks, the active participation of the researcher throughout the entire process is still crucial to maintain control over the quality, accuracy, and objectivity of their work.