JMIR Publications

ABSTRACT

Background:

Automated understanding of radiology reports has become increasingly important for clinical decision support, reg- istry construction, and the development of large language models (LLMs). However, most benchmark datasets are limited to English, hindering progress in multilingual clinical natural language processing (NLP).

Objective:

We present the NTCIR-18 RadNLP 2024 benchmark, a multilingual shared task for automated lung cancer TNM staging from radiology reports. Building upon three consecutive NTCIR editions, the benchmark provides Japanese and English radiology reports together with standardized evaluation protocols and community baselines.

Methods:

The NTCIR-18 RadNLP 2024 task attracted 19 participating teams from academia and industry. Participants developed systems using approaches ranging from conventional machine learning to state-of-the-art LLMs. We summarize the dataset, task design, participating methods, and benchmark results.

Results:

LLM-based methods substantially outperformed conventional approaches, although performance varied across TNM categories and languages. The benchmark also highlights persistent challenges in clinically faithful information extraction from radiology reports.

Conclusions:

Rather than serving solely as a shared-task report, the NTCIR-18 RadNLP 2024 benchmark establishes a reusable evaluation resource for multilingual clinical NLP. We anticipate that it will facilitate future research on structured information extraction, report generation, and clinically grounded evaluation of radiology LLMs.