Accepted for/Published in: Journal of Medical Internet Research
Date Submitted: Jun 25, 2024
Open Peer Review Period: Jul 2, 2024 - Aug 27, 2024
Date Accepted: Dec 11, 2024
(closed for review but you can still tweet)
Warning: This is an author submission that is not peer-reviewed or edited. Preprints - unless they show as "accepted" - should not be relied on to guide clinical practice or health-related behavior and should not be reported in news media as established information.
The performance of large language models in managing abnormal results of cervical cancer screening: Comparative Study
ABSTRACT
Background:
Cervical cancer remains the fourth leading cause of female death globally. Screening for cervical cancer is an effective preventative strategy. However, its impact is lessened in environments with scarce medical resources due to poor clinical decision-making and improper resource allocation. Large Language Models (LLMs) could significantly enhance medical systems in these settings by improving decision-making processes.
Objective:
This study aims to evaluate the performance of LLMs in managing abnormal cervical cancer screening results.
Methods:
Models are selected from AlpacaEval leaderboard version 2.0 and the capability of our computer. Questions inputted to models are designed in accordance to CSCCP and ASCCP guidelines. Two experts review the response from each model for accuracy, guideline compliance, clarity, and practicality by grading as A, B, C and D weighted as 3, 2, 1 and 0 scores, respectively. Effective rate is calculated as the ratio of the number of A and B to the number of all designed questions.
Results:
Nine models are included in this study, while 33 questions are specifically designed. Seven models (ChatGPT 4.0 Turbo, Claude 2, Gemini Pro, Mistral-7B-v0.2, Starling-LM-7B alpha, HuatuoGPT and BioMedLM 2.7B) provide stable responses. Among all included models, ChatGPT 4.0 Turbo and Claude 2 ranked in first level with mean score 2.30[2.0, 2.60] (effective rate: 88.81%) and 2.21[1.88, 2.54] (effective rate: 78.79%) when compared to the other seven models (P<0.001).
Conclusions:
Proprietary LLMs, particularly ChatGPT 4.0 Turbo and Claude 2, show promise in clinical decision-making involving logical analysis. However, this study underscores the need for further research to explore the practical application of LLMs in medicine.
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.