JMIR Publications

ABSTRACT

Background:

Patients often struggle with determining which outpatient specialist to consult based on their symptoms. Natural language processing models in healthcare offer the potential to assist patients in making these decisions before visiting a hospital.

Objective:

This study aims to evaluate the performance of ChatGPT in recommending medical specialties for medical questions.

Methods:

We utilized a dataset of 31,482 medical questions, each answered by doctors and labeled with the appropriate medical specialty from the health consultation board of NAVER, a major Korean portal. This dataset includes 27 distinct medical specialty labels. We compared the performance of the fine-tuned KM-BERT and ChatGPT models by analyzing their ability to accurately recommend medical specialties. We categorized responses from ChatGPT into those matching the 27 predefined specialties and those that did not. Both models were evaluated using performance metrics of accuracy, precision, recall, and F1-score.

Results:

ChatGPT demonstrated an answer avoidance rate of 6.2% but provided accurate medical specialty recommendations with explanations that elucidated the underlying pathophysiology of the patient’s symptoms. It achieved an accuracy of 0.939, precision of 0.219, recall of 0.168, and an F1-score of 0.134. In contrast, the KM-BERT model, fine-tuned for the same task, outperformed ChatGPT with an accuracy of 0.977, precision of 0.570, recall of 0.652, and an F1-score of 0.587.

Conclusions:

Although ChatGPT did not surpass the fine-tuned KM-BERT model in recommending the correct medical specialties, it showcased notable advantages as a conversational AI model. By providing detailed, contextually appropriate explanations, ChatGPT has the potential to significantly enhance patient comprehension of medical information, thereby improving the medical referral process.