Accepted for/Published in: Journal of Medical Internet Research
Date Submitted: Sep 1, 2023
Date Accepted: Apr 30, 2024
ChatGPT, Google Assistant: Which is best for patients with amblyopia?
ABSTRACT
Background:
We queried ChatGPT and Google Assistant about amblyopia and compared its answers to the keywords found in the website of American Association for Pediatric Ophthalmology and Strabismus, section for amblyopia. Of the 26 keywords chosen from website: ChatGPT included 11/26 in its responses while Google included 8/26.
Objective:
Our study investigates ChatGPT-3.5TM and Google assistantTM’s adherence to the American Association for Pediatric Ophthalmology and Strabismus’ (AAPOS) guidelines for patient education of amblyopia.
Methods:
ChatGPT-3.5 was used, available from November 30, 2022. The 4 questions were taken from American Association for Pediatric Ophthalmology and Strabismus’s website (AAPOS.org), specifically its glossary section for amblyopia: 1) What is amblyopia? 2) What causes amblyopia? 3) How is amblyopia treated? 4) What happens if amblyopia is untreated? Approved and selected by Dr. Gloria Wu and Dr. David Lee, the keywords from AAPOS.org were words or phrases that are deemed significant to the education of patients with amblyopia. The “Flesch-Kincaid Grade Level” formula, approved by the US Department of Education, was used to evaluate the reading comprehension level for the ChatGPT, Google Assistant, and AAPOS responses.
Results:
In their responses, ChatGPT did not mention the term “ophthalmologist” whereas Google Assistant and AAPOS.org both mentioned the term once, and twice respectively. ChatGPT did, however, use the term “eye doctors” once. From the Flesch- Kincaid test, AAPOS.org reading level was 11.4 ± 2.1, the lowest level. Google displayed 13.1 ± 4.8, the highest required reading level while also having the greatest variation in grade level in its responses. ChatGPT answers scored 12.4 ± 1.1. They were all similar in terms of level of difficulty in reading. For the key words, of the 4 responses, ChatGPT used the keywords 11/26 times, and Google Assistant 8/26 times.
Conclusions:
ChatGPT trains on texts and phrases and generates new sentences vs. Google Assistant automatically copies website links. As ophthalmologists, through our websites and journals, we should consider encoding “see an ophthalmologist” in our websites. While ChatGPT is here to stay, we, as physicians, need to monitor its answers.
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.