JMIR Publications

ABSTRACT

Background:

Large Language Models (LLMs) have demonstrated remarkable capabilities in natural language processing tasks, including medical question-answering (QA). However, individual LLMs often exhibit varying performance across different medical QA datasets, highlighting the need for strategies that can harness their collective strengths. Ensemble learning methods, which combine multiple models to improve overall accuracy and reliability, offer a promising approach to address this challenge. In this study, we introduce the LLM-Synergy framework, employing two ensemble methods—Boosting-based Weighted Majority Vote and Cluster-based Dynamic Model Selection—to enhance performance across diverse medical QA tasks.

Objective:

To enhance the accuracy and reliability of medical QA tasks by developing efficient ensemble learning approaches deploying the LLM technologies. We focus on improving performance across diverse medical QA datasets through innovative ensemble strategies.

Methods:

Our study employs three medical QA datasets: PubMedQA, MedQA-USMLE, and MedMCQA, each presenting unique challenges in biomedical QA. The proposed LLM-Synergy framework, focusing exclusively on zero-shot LLMs, incorporates two primary ensemble methods. The first is a Boosting-based weighted majority vote ensemble, where decision-making is expedited and refined by assigning variable weights to different LLMs through a boosting algorithm. The second method is Cluster-based Dynamic Model Selection, which dynamically selects the most suitable LLM votes for each query, based on the characteristics of question contexts, utilizing a clustering technique to optimize model selection..

Results:

Both the Majority Weighted Vote and Dynamic Model Selection methods demonstrate superior performance compared to individual LLMs across three medical QA datasets. Specifically, the accuracies are 35.84%, 96.21%, and 37.26% for MedMCQA, PubMedQA, and MedQA-USMLE, respectively, with the Majority Weighted Vote. Correspondingly, the Dynamic Model Selection yields slightly higher accuracies of 38.01% for MedMCQA, 96.36% for PubMedQA, and 38.13% for MedQA-USMLE.

Conclusions:

The LLM-Synergy framework utilizing two ensemble methods, represents a significant advancement in leveraging LLMs for medical QA tasks. This framework provides an innovative and efficient approach to utilizing LLM technologies, enabling customization for both current and potentially future challenges in biomedical and health informatics research.