JMIR Publications

ABSTRACT

Background:

With large language models (LLMs) like GPT-4 now widely accessible to clinicians via public web interfaces, it is crucial to evaluate their practical utility in real-world workflows. This study presents a real-world use evaluation of the GPT-4 web interface, simulating how clinicians might use this "off-the-shelf" tool to extract key diagnostic information from complex psychiatric electronic medical records (EMRs). Our objective was to assess the fundamental capabilities of this widely available tool in a zero-code, clinically-focused scenario, rather than conducting a technical evaluation requiring API development or model fine-tuning.

Objective:

This study aimed to systematically evaluate the diagnostic performance of GPT-4 in extracting key diagnostic information from real-world mental health clinical text data. Specifically, we intended to assess its accuracy in identifying core mental health disorders and compare its performance with the diagnostic conclusions of senior psychiatrists to verify its practical utility in clinical workflows.

Methods:

We conducted a comprehensive analysis using GPT-4 to classify EMRs into schizophrenia and varying levels of depression severity (mild, moderate, severe). The model's performance was evaluated using multiple metrics including accuracy, precision, recall, F1 score, specificity, Matthews correlation coefficient, and Cohen’s kappa score. Diagnostic efficacy and reliability were assessed by comparing GPT-4’s outputs with professional physician diagnoses.

Results:

The model demonstrated high accuracy (0.911) and precision (0.930) in distinguishing schizophrenia from depressive episode. However, its performance in determining depressive episode severity was weaker, with an accuracy of 0.496. Misclassification was most prevalent in mild and severe cases, which were often categorized as moderate depressive episode. GPT-4’s diagnostic discrepancies were attributed to symptom overlap and differences in subjective interpretation.

Conclusions:

The GPT-4 web interface demonstrates significant potential as an assistive tool for clinicians, capably extracting key diagnostic information from complex, unstructured psychiatric EMRs. However, the inherent limitations of a public web interface—namely the lack of reproducibility and parameter control—mean it should not be relied upon as a standalone diagnostic instrument. Its current value lies in its role as an efficient information extraction and summarization assistant for clinical workflows. Future research should progress beyond web interfaces to utilize controlled API environments, allowing for systematic evaluation, optimization, and the development of more reliable and clinically-integrated AI applications.