JMIR Publications

ABSTRACT

Background:

With the aging population, the burdens associated with vertebral compression fractures (VCFs) are expected to continue rising, making their comprehensive management a critical challenge in spinal surgery, necessitating adherence to authoritative, up-to-date clinical guidelines throughout the perioperative period and the implementation of standardized, multidimensional diagnostic and therapeutic strategies. However, both clinicians and patients struggle to access and track relevant information timely. The emergence of the large language models (LLMs) opens up the possibility to address this issue.

Objective:

This study aimed to evaluate the performance of LLMs, including DeepSeek-R1 and ChatGPT-5, in generating responses consistent with VCF clinical guidelines.

Methods:

With 2024 North American Spine Society (NASS) clinical guidelines for VCF as the reference standard, 34 open-ended questions and 87 closed-ended questions were input into DeepSeek-R1 (DeepSeek) and ChatGPT-5 (OpenAI). 4 senior spine surgeons independently evaluated the accuracy, consistency, self-awareness, and fabrication and falsification of the models’ responses using a 5-point Likert scale. Furthermore, the comprehensiveness, clarity, trust and satisfaction of open-ended questions were further evaluated. Subgroup analysis was conducted based on question type, recommendation grade, and VCF type, while simultaneously comparing model differences and calculating the interrater agreements.

Results:

A total of 726 responses were generated for 121 questions. The interrater agreements of the evaluators ranged from moderate to excellent. For closed-ended questions, ChatGPT-5 and DeepSeek-R1 demonstrated comparable performance in accuracy, fabrication and falsification. With regard to consistency, DeepSeek-R1 outperformed ChatGPT-5 in both closed-ended and open-ended questions. Furthermore, in response to open-ended questions, ChatGPT-5 and DeepSeek-R1 showed statistically significant differences in comprehensiveness, and trust and satisfaction, but no differences were observed in accuracy, self-awareness, fabrication and falsification, and clarity. Subgroup analysis showed that closed-ended questions overall outperformed open-ended questions. The responses to the questions with recommendation grade A–C outperformed those with recommendation grade I in accuracy, consistency, and fabrication and falsification, but performed lower in self-awareness. No statistically significant differences between VCF subtypes were observed across dimensions.

Conclusions:

LLMs demonstrated acceptable performance in answering questions from the VCF clinical guidelines, but their performance was influenced by factors such as recommendation grades, question types and sections. Although the current version is not yet fully reliable for clinical practice, future iterations of LLMs hold promise as dynamically updated guideline references.