JMIR Publications

ABSTRACT

Background:

This study explores the cutting-edge abilities of large language models (LLMs) such as ChatGPT in medical history taking and medical chart documentation, with a focus on their practical effectiveness in clinical settings—an area vital for the progress of medical artificial intelligence.

Objective:

The aim was to assess the capability of ChatGPT versions 3.5 and 4.0 in performing medical history intakes and charting in simulated clinical environments. The study compared the performance of non-medical individuals using ChatGPT with that of junior medical residents.

Methods:

A simulation involving standardized patients was designed to mimic authentic medical history-taking interactions. Five non-medical participants utilized ChatGPT versions 3.5 and 4.0 to conduct medical histories and document charts, mirroring the tasks performed by five junior residents in identical scenarios. A total of ten diverse scenarios were examined.

Results:

Evaluation of the medical documentation created by laypersons with ChatGPT assistance and by junior residents was conducted by two senior emergency physicians, employing audio recordings and the final charts. The assessment used the Objective Structured Clinical Examination (OSCE) benchmarks in Taiwan as a reference. ChatGPT 4.0 exhibited substantial enhancements over its predecessor and met or exceeded the performance of human counterparts in terms of both checklist and global assessment scores. Although the overall quality of human consultations remained higher, ChatGPT 4.0's proficiency in medical documentation was notably promising.

Conclusions:

The performance of ChatGPT 4.0 was on par with human participants in OSCE evaluations, signifying its potential in medical history documentation and chart writing. Despite this, the superiority of human consultations in terms of quality was evident. The study underscores both the promise and the current limitations of LLMs in the realm of clinical practice.