JMIR Publications

ABSTRACT

Background:

Large Language Models (LLMs) like OpenAI's GPT-4 are increasingly utilized in healthcare for automated medical consultation, yet there has been limited investigation into their performance in comparison to human experts.

Objective:

To compare the medical accuracy of GPT-4 with human experts in providing medical advise using real-world user-generated queries.

Methods:

A panel of three independent cardiologists was engaged to evaluate responses from both human experts and GPT-4. A dataset of 251 cardiology-specific question-answer pairs was collected from an Internet portal. The evaluation focused on assessing the medical accuracy of the responses. Additionally, the linguistic evaluation encompassed examining the length and vocabulary diversity of answers provided by GPT-4 compared to human experts.

Results:

Both GPT-4 and human experts demonstrated similar levels of medical accuracy. Rather, the proportion of low-accuracy answers was higher among human experts (0.4% vs. 4.6%). GPT-4's responses were generally longer and utilized less diverse vocabulary, potentially enhancing their comprehensibility for general users. Nevertheless, human experts outperformed GPT-4 in specific question categories, notably those related to drug/medication information and preliminary diagnoses. These findings highlight the limitations of GPT-4 in providing advice based on clinical experience.

Conclusions:

GPT-4 has shown promising potential in automated medical consultation, with comparable medical accuracy to human experts. However, challenges remain, particularly in the realm of nuanced clinical judgement. Future improvements in LLMs may require the integration of specific clinical reasoning pathways and regulatory oversight for safe use. Further research is needed to understand the full potential of LLMs across various medical specialties and conditions.