Accepted for/Published in: JMIR Research Protocols
Date Submitted: Feb 1, 2024
Date Accepted: Apr 10, 2024
Warning: This is an author submission that is not peer-reviewed or edited. Preprints - unless they show as "accepted" - should not be relied on to guide clinical practice or health-related behavior and should not be reported in news media as established information.
Assessing and Optimizing Large Language Models on Spondyloarthritis Multi-choice Question Answering: Protocol for a Bilingual Evaluation Benchmark
ABSTRACT
Background:
Spondyloarthritis (SpA), a chronic inflammatory disorder, predominantly impacts the sacroiliac joints and spine, significantly escalating the risk of disability. SpA's complexity, as evidenced by its diverse clinical presentations and symptoms that often mimic other diseases, presents substantial challenges in its accurate diagnosis and differentiation. This complexity becomes even more pronounced in non-specialist healthcare environments due to limited resources, resulting in delayed referrals, increased misdiagnosis rates, and exacerbated disability outcomes for SpA patients. The emergence of large language models (LLMs) in medical diagnostics introduces a revolutionary potential to overcome these diagnostic hurdles. Despite recent advancements in AI and LLMs demonstrating effectiveness in diagnosing and treating various diseases, their application in SpA remains underdeveloped. Presently, there is a notable absence of SpA-specific LLMs and an established benchmark for assessing the performance of such models in this particular field.
Objective:
Our objective is to develop a foundational medical model, creating a comprehensive evaluation benchmark tailored to the essential medical knowledge of spondyloarthritis (SpA) and its unique diagnostic and treatment protocols. The model, post pre-training, will be subject to further enhancement through supervised fine-tuning. It is projected to significantly aid physicians in SpA diagnostics and treatment, especially in settings with limited access to specialized care. Furthermore, this initiative is poised to promote early and accurate SpA detection at the primary care level, thereby diminishing the risks associated with delayed or incorrect diagnoses.
Methods:
A rigorous benchmark, comprising 222 meticulously formulated multiple-choice questions on spondyloarthritis, will be established and developed. These questions will be extensively revised to ensure their suitability for accurately evaluating LLMs' performance in real-world diagnostic and therapeutic scenarios. Our methodology involves selecting and refining top foundational models using public datasets. The best-performing model in our benchmark will undergo further training. Subsequently, over 80,000 real-world inpatient and outpatient cases from hospital will enhance LLM training, incorporating techniques such as Supervised Fine-Tuning and Low-Rank Adaptation. We will rigorously assess the models' generated responses for accuracy and evaluate their reasoning processes using metrics of fluency, relevance, completeness, and medical proficiency.
Results:
Development of the model is progressing, with significant enhancements anticipated by early 2024. The benchmark, along with the results of evaluations, is expected to be released in the second quarter of 2024.
Conclusions:
Our trained model aims to capitalize on the capabilities of LLMs in analyzing complex clinical data, thereby enabling precise detection, diagnosis, and treatment of SpA. This innovation is anticipated to play a vital role in diminishing the disabilities arising from delayed or incorrect SpA diagnoses. By promoting this model across diverse healthcare settings, we anticipate a significant improvement in SpA management, culminating in enhanced patient outcomes and a reduced overall burden of the disease.
Citation
