JMIR Publications

ABSTRACT

Background:

Psychiatric disorders are common but diagnostically complex, with heavy dependence on the clinical experience of physicians. In cases of a shortage of medical resources, ensuring accurate and timely diagnoses becomes particularly challenging. Recently, large language models (LLMs) have exhibited promising potential in simulated datasets, offering a new technological path for diagnosing psychiatric disorders. However, their overall performance remains insufficiently evaluated in real-world clinical settings.

Objective:

This study aims to systematically evaluate the diagnostic performance of large language models (LLMs) in real-world psychiatric clinical settings. By integrating electronic health records from six provincial and municipal psychiatric centers in China, we compared the diagnostic capabilities of GPT-4.0, GPT-3.5, and the Chinese model GLM-4-Plus across different age groups and psychiatric disorder categories. The objective is to explore how LLMs adapt to diverse patient populations and psychiatric conditions in practical clinical applications, providing evidence for the utility of LLMs in intelligent diagnostic support for mental disorders.

Methods:

We conducted a retrospective study involving the electronic health records (EHRs) of 9,923 psychiatric inpatients from six provincial and municipal-level psychiatric centers in China. The data were collected between July 2017 and July 2024. The dataset covered all psychiatric disorder categories in ICD-10 (F0–F9). We evaluated the diagnostic capabilities of three representative LLMs, namely GPT-4.0, GPT-3.5, and GLM-4-Plus. Experienced psychiatrists confirmed standard diagnoses. Precision, recall, and F1-score were used to assess model outputs. Stratified comparisons were made based on the disorder type and age group.

Results:

The diagnostic accuracy of the LLMs was the highest in the elderly patient group (74.3%–79.5%). However, the accuracy was lower in the adolescent and young adult groups (53.7%–61.0% and 66.4%–70.0%, respectively). Between-group comparisons revealed statistically significant differences in the scores between the adolescent, middle-aged, and elderly groups and between the young adult and middle-aged and elderly groups (P < 0.001 for all comparisons). Among all models, GPT-4.0 exhibited the best diagnostic performance (overall accuracy = 71.7% and weighted average F1 score = 0.881), outperforming GPT-3.5 (68.8%, F1 = 0.849) and GLM-4-Plus (69.3%, F1 = 0.873). The performance of GPT-4.0 was the best in high-prevalence categories such as mood disorders (F3) and schizophrenia spectrum disorders (F2). However, all three models exhibited varying performance fluctuations in lower-prevalence categories.

Conclusions:

Patient age significantly affects the performance of LLMs in diagnostic assistance, with elderly patients exhibiting higher diagnostic accuracy compared with younger patients. GPT-4.0 exhibits outstanding accuracy and stability in identifying psychiatric disorders, effectively helping physicians in decision-making in clinical settings, decreasing cognitive load, and improving the overall efficiency and standardization of psychiatric diagnostic processes. Clinical Trial: Psychiatric disorders; Large language models; Artificial intelligence; Multicenter study; Real-world data