JMIR Publications

ABSTRACT

Background:

The integration of artificial intelligence into oncology promises to augment clinical decision-making, yet the practical utility of LLMs in complex subspecialties like rectal cancer is not well-defined. Current benchmarks often overlook the critical differences between general and medical-optimized models in handling intricate treatment algorithms involving staging and immunotherapy biomarkers.

Objective:

To assess the diagnostic and therapeutic decision-making capabilities of seven LLMs using a cohort of simulated rectal cancer cases, and to delineate the specific limitations and contextual dependencies that must be addressed prior to clinical deployment.

Methods:

We developed 60 virtual cases covering various stages and molecular profiles of rectal cancer. Seven LLMs (five general, two medical) were assessed. Their generated treatment plans were blindly reviewed by senior oncologists and scored for staging accuracy, treatment appropriateness, and execution details.

Results:

Inter-rater reliability was excellent (Kappa coefficient range 0.68–0.84), ensuring assessment robustness. Analysis revealed that general-purpose models like GPT-5.2 and Gemini-3.1 significantly outperformed specialized medical-enhanced models in staging accuracy tasks. For treatment decision appropriateness, DeepSeek-V3.2 and Gemini-3 demonstrated the best and most stable performance, while GPT-5.2 led in the standardization of treatment execution. Subgroup analysis uncovered critical limitations: all models faced significant challenges in formulating treatment strategies for locally advanced cases (especially stage III C), and their performance was inconsistent regarding immunotherapy decisions related to dMMR subtypes, highlighting a pronounced "uneven proficiency" and high context-dependency of model capabilities.

Conclusions:

Leading general-purpose LLMs show potential for certain aspects of rectal cancer diagnosis and treatment, though their performance varies across tasks, disease stages, and molecular subtypes. Future work should focus on optimizing and validating these models for complex clinical scenarios. Integration into practice must proceed cautiously to ensure safe, effective, and equitable AI-assisted decision-making.