JMIR Publications

ABSTRACT

Background:

ChatGPT has shown great potential in clinical diagnosis and could become an excellent auxiliary tool in clinical practice. In our study, we investigate and evaluate ChatGPT in diagnostic capabilities by comparing the performance of GPT-4.0 and GPT-3.5 across model iterations.

Objective:

This study aims to evaluate the precise diagnostic ability of ChatGPT for colon cancer and its potential as an auxiliary diagnostic tool for surgeons.

Methods:

We conducted a prospective study to investigate whether GPT-4.0 or GPT-3.5 can provide an accurate diagnosis based on clinical information from the public case database. The results were analyzed regarding items’ specialty classification and error patterns (medical histories, symptoms, physical signs, examination results, and intraoperative findings).

Results:

For primary diagnose, the accuracy rates of GPT-4.0 were significantly higher than GPT-3.5 (0.972±0.137 vs. 0.855±0.335, p<0.001). For secondary diagnoses, the accuracy rates of GPT-4.0 were also significantly higher than GPT-3.5 (0.908±0.159 vs. 0.617±0.349, p<0.001). GPT-3.5 has limitations in processing patient history, symptom presentation, laboratory tests, and imaging data effectively. GPT-4.0, while improving upon GPT-3.5, still has limitations in identifying symptoms and laboratory test data by classifying and analyzing the causes of misdiagnosis. For both primary and secondary diagnoses, there was no significant difference in age, sex, and system group in GPT-4.0 and GPT-3.5 (p>0.05).

Conclusions:

Based on the findings of this study, it is evident that ChatGPT has potential in the field of medical diagnosis. The diagnostic accuracy of GPT-4.0 is better than that of GPT-3.5, but GPT-4.0 still has limitations regarding patient symptoms and laboratory data recognition. Further studies performed in the dynamic clinical practice environment are needed.