JMIR Publications

ABSTRACT

Background:

Outpatient patients presenting with low back pain (LBP), one of many common musculoskeletal disorders (MSDs), often require efficient preconsultation triage and early differential diagnosis support. Large language models (LLMs) may assist with these tasks in text-based clinical scenarios.

Objective:

To compare the performance of ChatGPT 5.1 and DeepSeek V3.2 in musculoskeletal triage and the differential diagnosis of outpatients with LBP using real-world outpatient records under two simulated information conditions (chief complaint only vs. structured questionnaire).

Methods:

This retrospective comparative study was conducted at a tertiary academic teaching hospital in Beijing. After screening outpatients attending the orthopedic clinic between November 1, 2024 and December 31, 2024, 160 cases were included across 8 diagnostic categories (20 per category): 6 musculoskeletal etiologies (lumbar disc herniation, lumbar spinal stenosis, ankylosing spondylitis, osteoporotic vertebral compression fracture, infectious diseases of the spine, and metastatic spinal tumor) and 2 nonmusculoskeletal conditions (multiple myeloma and urinary system diseases). Two orthopedic surgeons established an expert reference standard with structured adjudication. Evaluation was performed in two phases: Phase I (chief complaint) and Phase II (structured questionnaire with 7 domains/33 items), both executed in a zero-shot setting using standardized prompts. Outcomes included (1) MSD identification accuracy, (2) primary diagnosis accuracy, and (3) differential diagnosis agreement (0–3 correct differentials). During Phase II, three senior physicians additionally rated model rationales on four domains using a 5-point Likert scale. Interrater agreement was assessed via intraclass correlation coefficients (ICCs). Statistical comparisons were performed using McNemar and Mann–Whitney U tests.

Results:

The cohort included 160 cases (84/76 male/female). For MSD identification, DeepSeek V3.2 and ChatGPT 5.1 had an overall accuracy of 85% and 81% in Phase I, which improved to 91% and 94% in Phase II, respectively. For primary diagnosis, overall accuracy of DeepSeek V3.2 and ChatGPT 5.1 increased from 48% and 35% in Phase I to 77% and 88% in Phase II, respectively, with between-model differences and within-model improvements reported as statistically significant. Mean correct differential diagnoses for DeepSeek V3.2 and ChatGPT 5.1 increased from 1.28 (standard deviation (SD) 0.72) and 1.23 (SD 0.75) in Phase I to 2.01 (SD 0.73) and 2.02 (SD 0.76) in Phase II, respectively (p < 0.05). In Phase II rationale ratings, mean scores were generally “good,” with ChatGPT 5.1 scoring higher in understanding/reasoning (p < 0.05). ICCs indicated moderate-to-good agreement across domains.

Conclusions:

Both LLMs demonstrated clinically promising text-based triage and differential diagnosis support for LBP, with substantial performance gains when structured clinical information was provided. ChatGPT 5.1 showed stronger reasoning with structured inputs, whereas DeepSeek V3.2 performed comparably in chief complaint triage. These findings highlight the need for further validation in prospective, multicenter studies and the development of higher-fidelity workflows. Clinical Trial: This study was conducted in accordance with the ethical principles stated in the Declaration of Helsinki and was approved by our institutional ethics committee (2025-KE-417).