JMIR Publications

ABSTRACT

Background:

Large language models (LLMs) such as ChatGPT, Gemini and Claude are increasingly used by clinicians, yet their accuracy, safety and consistency in clinical cases remain poorly defined. Most studies assess LLMs using multiple-choice questions rather than free-response reasoning.

Objective:

This study aims to evaluate the performance and safety of three widely used LLMs in realistic vascular surgery scenarios to assess their fitness-for-use in clinical practice, and to elucidate the barriers that exist to their widespread integration.

Methods:

Forty-two fictitious vascular cases across five major pathologies: acute limb ischaemia (ALI), aortic disease (Ao), chronic limb ischaemic (CLI), diabetic foot infection (DFI) and extracranial cerebrovascular disease (ECD), were independently entered into ChatGPT-5, Gemini 2.5 and Claude Sonnet 4.5 using standardised prompts. Each model answered structured questions covering diagnosis, investigation and management (both operative and non-operative). Responses were scored by a panel /20 using a predefined rubric. Each plan’s overall safety was separately assessed. Comparative analyses utilised t-tests, ANOVA and multivariable logistic regression.

Results:

Mean composite scores by model were 86.5% (ChatGPT), 83.5% (Gemini), and 88.0% (Claude), whilst scores by disease were 82.7% (ALI), 83.8% (Ao), 85.9% (CLI), 86.1% (DFI) and 80.7% (ECD) (p=non-significant). Unsafe plans occurred in 11.9% (ChatGPT), 23.8% (Gemini) and 7.1% (Claude). On multivariable analysis, independent predictors of unsafe outputs were lower composite score (OR 0.47, p=0.001), higher word count (OR 1.003, p=0.001) and ALI (OR 20.9, p<0.001).

Conclusions:

Our findings demonstrate LLMs’ promise in managing routine vascular surgery cases. However, their inconsistent safety profiles and ethical limitations preclude unsupervised clinical use. Rigorous specialty-specific validation is essential before they may be integrated into routine practice.