JMIR Publications

ABSTRACT

Background:

Systematic literature reviews (SLRs) are essential for evidence synthesis in health research but remain labor-intensive, especially at the screening stage. Manual review of titles and abstracts requires substantial human effort, while existing automation tools still have limited adoption in health technology assessment. The EQ-5D questionnaire, a widely used patient-reported outcome measure for health-related quality of life, provides data that frequently underpin reimbursement and policy decisions.

Objective:

This pilot study evaluated whether recent large language models (LLMs) can automate the identification of publications reporting EQ-5D data in PubMed records, using only publicly available metadata (title, abstract, and keywords).

Methods:

A total of 200 publications retrieved through the EuroQol PubMed filter were manually labeled by experts as reporting or not reporting EQ-5D data. The dataset was split into stratified training, validation, and test subsets. Several machine learning approaches were compared, including a Naïve Bayes baseline using bag-of-words features, a decision-tree model based on full-text keyword occurrence, and transformer-based large language models (BERT, BioBERT, SciBERT, BlueBERT). Both classifier-only and fine-tuned configurations were tested across multiple learning rates. Model performance was assessed using accuracy, precision, recall, and F1 score.

Results:

Baseline approaches achieved near-random test performance (accuracy around 0.53). Classifier-only LLMs modestly improved results (accuracy up to 0.64 with SciBERT). Fine-tuned models substantially outperformed these baselines, with BioBERT achieving the best performance (accuracy = 0.71, F1 = 0.79). The models reproduced human screening tendencies despite the small dataset size, demonstrating the technical feasibility of LLM-assisted article selection.

Conclusions:

This study provides the first demonstration of LLM-based automation of EQ-5D data identification in biomedical literature. Although limited by dataset size, the proposed workflow is reproducible and adaptable to other patient-reported outcome measures. Future work will scale data collection, include statistical testing, and explore semi-supervised learning to further reduce manual screening workload.