Warning: This is an author submission that is not peer-reviewed or edited. Preprints - unless they show as "accepted" - should not be relied on to guide clinical practice or health-related behavior and should not be reported in news media as established information.
Improving Medical Question Answering in Online Health Consultations Through Retrieval-Augmented Semantic Aggregation of Multiple Physician Responses
ABSTRACT
Background:
Large language models (LLMs) have shown promise for medical question answering, but existing systems often underuse the diverse and complementary information contained in the common “one question, multiple answers” setting, in which multiple physicians respond to the same patient query.
Objective:
This study aimed to develop and evaluate a retrieval-augmented medical question answering framework enhanced by MedSAGE, a Medical Semantic Aggregation for Generative Question Answering module, to better synthesize multiple physician responses and improve the coherence, completeness, and clinical relevance of system-generated answers.
Methods:
We developed a retrieval-augmented generation framework incorporating MedSAGE, a medical semantic aggregation module that jointly models sentence-level consensus, query relevance, and semantic diversity across physician responses. The framework combines dense vector retrieval with structured semantic aggregation to transform fragmented multi-physician inputs into unified medical answers. We evaluated the framework on the publicly available cMedQA2 dataset, which contains more than 120,000 real-world medical question-answer pairs across multiple clinical departments. Performance was assessed using automated natural language generation metrics, terminology-specific analyses, physician ratings, and ablation experiments across multiple large language models.
Results:
The proposed framework consistently outperformed both base generation and standard retrieval-augmented generation across the evaluated models. For Qwen-32B, BLEU-2 increased from 1.10% to 13.35%, BERTScore-F1 increased from 57.05% to 70.99%, METEOR increased from 19.63% to 55.54%, and CIDEr increased from 7.04% to 24.44%. In the ablation analysis, Distinct-2 increased from 6.55% to 9.46%. Physician evaluation further showed improved answer quality, with the highest answer accuracy reaching 4.08 out of 5.0, alongside gains in actionability and overall satisfaction. The framework also improved the coverage of key medical terminology categories, including disease, drug, treatment, and examination entities.
Conclusions:
MedSAGE-enhanced semantic aggregation can strengthen retrieval-augmented medical question answering by transforming fragmented physician responses into more coherent, complete, and clinically meaningful outputs. This framework may support digital health consultation systems and related medical informatics applications that require structured synthesis of heterogeneous physician-generated responses. Clinical Trial: Not applicable.
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.