JMIR Publications

ABSTRACT

Background:

Artificial intelligence (AI) models are being increasingly integrated into clinical care. Moreover, the accessibility of open-AI resources makes them attractive to patients seeking clinical information. Little is known regarding the utility of large language models as patient resources for navigating major cancer diagnoses.

Objective:

This study evaluated the content, readability and safety of ChatGPT 4.o-generated responses to common perioperative queries about hepatic, pancreatic, and colon cancers.

Methods:

A 28-question survey was developed based on frequently asked surgical questions for select malignancies. Surgical oncologists rated ChatGPT 4.o-generated responses on a 5-point Likert Scale for accuracy, quality, and tangibility. Readability was assessed using the Flesch-Kincaid Reading Grade Level (FKRGL). Respondents provided free-text comments and reported comfort with patients using ChatGPT. Survey completion implied consent.

Results:

Seven attending surgical oncologists with a median of 7 years in practice completed the survey. Responses received mean scores of 3.5/5 for quality, 3.6/5 for accuracy, and 3.6/5 for tangibility. The responses had a median FKRGL score of 14.5. On post-hoc analysis for select questions, median FKRGL was 15.6, decreasing to 7.1 and 14.5 with prompting and rephrasing. Numerous inaccuracies and content gaps were reported, and approximately 43% of providers did not report feeling “comfortable” in having patients consult publicly available AI for medical information.

Conclusions:

This study provides cautionary, yet optimistic, findings regarding the value of open-source ChatGPT as a patient resource for abdominal malignancies. Providers should be prepared to effectively counsel patients concerning use of ChatGPT to mitigate minor inaccuracies and address readability challenges.