JMIR Publications

ABSTRACT

Background:

Multimodal large language models (MLLMs) are emerging as powerful tools in medical diagnostics, with the potential to interpret both images and text simultaneously. However, their practical performance in orthopedics—particularly in diagnosing complex femoral head necrosis—and the optimal methods for integrating them into clinical workflows remain under-explored.

Objective:

This study aimed to evaluate the performance of different MLLMs in the diagnosis and staging of osteonecrosis of the femoral head (ONFH), and to examine how multimodal prompt design influences their translational potential within a human–AI collaborative workflow.

Methods:

This retrospective observational diagnostic study included 329 patients (658 hips) who underwent radiographs or MRI between July 2023 and December 2024 at a tertiary referral center. Four advanced MLLMs (ChatGPT-4o, Claude 3.7, Qwen2.5-VL, and Gemma-3) were evaluated using three classification systems (Ficat, ARCO, Steinberg). Three prompt configurations were tested: single image (SI), image plus structured description (ID), and multi-image (MI). Primary outcomes were diagnostic accuracy (AUC, F1 score), staging accuracy, grading reliability (intraclass correlation coefficient [ICC]).

Results:

Model performance was highly dependent on prompt design. SI input yielded poor results (mean detection AUC, 0.52). ID input significantly improved performance, with AUC 0.91, staging accuracy 0.77, and grading reliability 0.96, achieving clinically comparable performance to junior orthopedic surgeons. MI input provided no additional benefit, underscoring current limitations in multi-image reasoning. Open-source models performed similarly to commercial counterparts but used different diagnostic strategies.

Conclusions:

When integrated into a human–AI collaborative workflow, MLLMs demonstrate potential as assistive tools for ONFH diagnosis and staging, particularly for supporting standardized interpretation and grading consistency. Optimizing multimodal prompt design and enhancing open-source model performance could accelerate the clinical adoption of MLLMs, advance clinical decision support, and demonstrate the translational potential of MLLMs within orthopedic workflows.