JMIR Publications

ABSTRACT

Background:

Artificial intelligence (AI), particularly large language models (LLMs), has shown potential across various domains, but their performance in multidisciplinary teams (MDT) remains largely unknown.

Objective:

This study aimed to evaluate LLMs' performance in supporting radiologists within multidisciplinary breast cancer teams, focusing on their roles in clinical decisionmaking and improving patient care.

Methods:

A set of 50 questions related to radiological and breast cancer guidelines was developed to assess breast cancer. These questions were posed to nine LLMs (ChatGPT-4.0, ChatGPT-4o, ChatGPT-4o mini, Claude 3 Opus, Claude 3.5 Sonnet, Gemini 1.5 Pro, Tongyi Qianwen 2.5, ChatGLM, Ernie Bot 3.5) and clinical physicians. Responses were evaluated for accuracy, confidence, and consistency, based on the 2023 NCCN Breast Cancer Guidelines and 2013 ACR BI-RADS recommendations.

Results:

Claude 3 Opus and ChatGPT-4 achieved the highest confidence scores (2.78 and 2.74, respectively). For consistency, Claude 3 Opus and Claude 3.5 Sonnet led with scores of 3.0, followed by ChatGPT-4o and Gemini 1.5 Pro. ChatGPT-4o mini excelled in clinical diagnostics with a score of 3.0, outperforming physician groups.ChatGPT-4 also scored higher than physicians across several areas, while ChatGLM and Ernie Bot 3.5 underperformed. Attending physicians and residents scored similarly, with fellows showing slightly lower scores in radiotherapy.

Conclusions:

LLMs like ChatGPT-4o and Claude 3 Opus show promise for breast cancer diagnostics and MDT support, but they cannot fully replace clinical experience in complex cases. Further AI refinement is needed for clinical applicability.