Accepted for/Published in: JMIR Cancer
Date Submitted: Oct 17, 2025
Open Peer Review Period: Oct 20, 2025 - Dec 15, 2025
Date Accepted: Mar 5, 2026
(closed for review but you can still tweet)
Evaluation of a RAG-Based LLM for Evidence-Based Herb and Supplement Information in Cancer Care
ABSTRACT
Background:
Large language models (LLMs) are increasingly used to seek information about herbs and dietary supplements in cancer care. However, these models are prone to hallucinations, which can be harmful in oncology settings.
Objective:
To develop and evaluate the performance of a retrieval augmentation generation (RAG)-based LLM in answering common oncology provider questions about herbs and dietary supplements.
Methods:
We developed a RAG-based LLM, grounded in the expert-curated About Herbs database that contains evidence-based and objective information on more than 300 herbs and dietary supplements. We compared the performance of our RAG-based LLM with nine popular LLMs in answering common oncology provider questions about herbs and dietary supplements.
Results:
Our model outperformed the nine LLMs in correctness (92.31% cases vs 61.54%) and conciseness (100% cases vs 65.38%) of responses to oncology provider questions about herbs and supplements. It slightly underperformed in completeness (84.62% vs 88.46%) likely because of the restricted knowledge source.
Conclusions:
This study found that a RAG-based LLM, grounded in evidence-based information, helped improve the quality and accuracy of responses to provider questions about herbs and dietary supplements, with no hallucinations.
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.