JMIR Publications

ABSTRACT

Background:

Literature review relies on rigorous title and abstract screening by researchers, which is time-consuming. Artificial intelligence (AI)-assisted literature screening tools have been proposed to improve efficiency by prioritising titles and abstracts with the highest likelihood of meeting the inclusion criteria, thereby reducing the need to screen all records. By examining AI-assisted screening approaches, this study contributes to understanding the practical applicability of AI-assisted tools in dementia care literature reviews.

Objective:

This study aims to assess the performance of two approaches within the AI-assisted screening tool ASReview compared to manual screening, in a review focused on how AI can support the quality of life of people with dementia.

Methods:

This study used a dataset of 4,690 titles and abstracts from a scoping review on AI and quality of life in dementia. The manual screening results served as the reference standard. Approach A used a simpler model with minimal prior input, whereas approach B used a more advanced model with a larger training set. Each approach applied a stopping rule requiring at least 10% of the dataset to be screened and 50 consecutive irrelevant abstracts. Performance was evaluated in terms of sensitivity, specificity, precision, accuracy, and screening time. Agreement between manual and ASReview approaches was assessed using Cohen’s kappa, and differences in how the two methods classified abstracts were examined with McNemar’s test. Performance and agreement were calculated at two levels: (1) after abstract screening, using the number of abstracts selected for full-text review in the manual approach, and (2) after full-text inclusion, using the number of final included studies as the reference standard.

Results:

Manual screening identified 283 abstracts for full-text review and included 30 final studies, requiring 19 hours. Approach A screened 830 abstracts (0.17) in 4.3 hours, with a sensitivity of 0.16 (Level 1) and 0.53 (16 abstracts at Level 2), and a precision of 0.62. Approach B screened 798 abstracts (0.17) in 5.5 hours, with a sensitivity of 0.23 (Level 1) and 0.70 ( 21 abstracts at Level 2), and a precision of 0.76. Both ASReview approaches showed high specificity (>0.98) and accuracy (>0.98). Agreement with the manual approach was moderate, with Cohen’s kappa values ranging from 0.24 to 0.35 across both levels. McNemar’s tests revealed a significant directional imbalance (P<.001), indicating that ASReview approaches were more likely to miss relevant abstracts included by the manual approach than to incorrectly include irrelevant abstracts.

Conclusions:

ASReview demonstrates significant potential for reducing the workload in systematic reviews in dementia care. Although approach B performed best, it still missed 30% of the final inclusions. To reduce the risk of missing relevant abstracts, researchers could add hard-to-detect abstracts manually to the training set and selectively review excluded abstracts to further improve recall while maintaining efficiency.