Previously submitted to: JMIR AI (no longer under consideration since Jan 28, 2025)
Date Submitted: Nov 10, 2024
Warning: This is an author submission that is not peer-reviewed or edited. Preprints - unless they show as "accepted" - should not be relied on to guide clinical practice or health-related behavior and should not be reported in news media as established information.
Evaluating the Use of Generative Artificial Intelligence in Academic Writing for Surgery and Urology: Instrument Validation Study
ABSTRACT
Background:
The use of artificial intelligence (AI) has gained rapid uptake in academic, scientific, and medical writing. While it can be helpful in summarising data and providing opinions, some studies have demonstrated limitations in accuracy and potential bias.
Objective:
This study aims to assess the accuracy of academic writing in the context of general surgery and urology topics and their suitability for use in scientific publications.
Methods:
20 subject matter experts provided 20 questions each for 9 General Surgery and Urology subspecialties. The questions were further classified into 4 categories: epidemiology, pathophysiology, clinical management, and surgical techniques. These questions were submitted to ChatGPT v3.5 requesting for 10 answers for each question using the following prompt “Write me 10 points on XXX, for submission to JAMA medical journal for publication, providing different reference for each point”. The primary outcomes rated by subject matter experts were: accuracy and breadth of the answers; language used in medical writing; number and accuracy of the references. Secondary outcome measured was the ability to detect AI-generated text using Giant Language-model Test Room (GLTR).
Results:
There were 180 questions posed to ChatGPT, with 120 General surgery (68.6%) and 60 Urology questions (31.4%). Overall accuracy of information provided in the answers was 1560/1800 (86.7%), with good overall breadth 163/180 (90.5%) and language 167/180 (92.7%). Answers provided for General Surgery were significantly more accurate than Urology (90.0% vs 79.6%, p<0.001). Only 5.5.% of references provided were true published references, with the majority being fictitious. Subgroup analysis showed a statistical difference in accuracy between the 4 categories of epidemiology (402/405, 89.3%), pathophysiology (404/450, 89.8%), clinical management (387/450, 86.0%), and surgical techniques (370/450, 82.2%) questions (p<0.001). For the secondary outcomes, frac(p) was 0.994, indicating that almost all statements were identified to be AI generated.
Conclusions:
While the information provided in ChatGPT is accurate with sufficient breadth and with appropriate medical academic language, its quotation of fictitious references limits its use for academic writing.
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.