Previously submitted to: JMIR Cancer (no longer under consideration since Sep 30, 2024)
Date Submitted: Aug 1, 2023
Open Peer Review Period: Jul 31, 2023 - Sep 25, 2023
(closed for review but you can still tweet)
Warning: This is an author submission that is not peer-reviewed or edited. Preprints - unless they show as "accepted" - should not be relied on to guide clinical practice or health-related behavior and should not be reported in news media as established information.
Exploring ChatGPT's potential in the context of colon cancer patient education: A comparative analysis
ABSTRACT
Background:
ChatGPT is a large language model capable of generating human-like conversation. It has demonstrated promise as a tool for medical education for both professionals and patients. Previous research in medical oncology and colon cancer showed a glimpse of its application on topics like colonoscopy, colorectal surgery, and guideline-based treatment.
Objective:
To evaluate ChatGPT's performance as a source of patient medical education for colon cancer
Methods:
A set of twenty non-expert questions were prepared and fed to ChatGPT three times. Later, generated responses were evaluated by two doctors for accuracy, simplicity (0-10), and consistency (0,1). Mean, median and standard deviation were calculated for both accuracy and simplicity scores along with the Intraclass Correlation Coefficient and confidence interval for inter-rater agreement assessment. For consistency, rate, cohen's kappa, standard error, and confidence interval were calculated.
Results:
Accuracy: Mean = 8.4, Median = 8.5, SD = 1.7. ICC: Avg. measures for absolute agreement = 0.7 (95% CI 0.25 to 0.88), for consistency = 0.74 (95% CI 0.34 to 0.9). Simplicity: Mean = 8.55, Median = 9, SD = 1.69. ICC: Avg. measures for absolute agreement = 0.65 (95% CI 0.12 to 0.86), for consistency = 0.72 (95% CI 0.28 to 0.89). Consistency: rate = 67.5%, Cohen's Kappa: 0.66 (SE = 0.18, 95% CI 0.31 to 1.0).
Conclusions:
In this study, we assessed ChatGPT's capabilities of answering patients' questions about colon cancer. Findings showed significant and promising results of answers' accuracy, simplicity, and consistency in multiple trials. However, there is room for improvement. As ChatGPT continues to gain popularity among users, research studies on the impact of this technology on patient outcomes are needed urgently to guide clinical application.
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.