JMIR Publications

ABSTRACT

Background:

Vision-language models (VLMs) for radiology require large-scale image–text pairs. However, free-text reports mix background information, findings, and continuation sentences. Manual annotation is labor-intensive, and the direct use of clinical reports raises privacy concerns.

Objective:

We aimed to develop a context-aware sentence classification model for Japanese radiology reports using synthetic and automatically annotated data and validate it using multi-institutional clinical reports.

Methods:

Synthetic Japanese radiology reports were generated using OpenAI API (GPT-4.1); sentence-level annotations were performed using GPT-4.1-mini in four categories: context, positive findings, negative findings, and continuation. After filtering, 3,104 reports were divided into training (2,670), validation (334), and testing (100) sets. For external validation, 280 reports dated October 1, 2024, were sampled from seven institutions in the Japan Medical Image Database and annotated by two radiologists. Large language models (Qwen3 and LLaMA 3.2) and Japanese text classification models (BERT base Japanese and ModernBERT-Ja-130M, JMedRoBERTa) were fine-tuned and evaluated for accuracy, macro–F1, and positive predictive value for label 1 (PPV_1).

Results:

For the internal test set (1,124 sentences), all models performed well: accuracy, 0.939–0.950; macroF1, 0.924–0.940; and PPV_1, 0.904–0.953. For the external dataset (3,477 sentences), the accuracy declined to 0.783–0.812 and macro–F1 to 0.761–0.790. Qwen3-4B showed the best performance (PPV_1 = 0.952).

Conclusions:

The model trained solely on synthetic reports showed robust performance in real-world Japanese radiology reports. This approach enables the efficient extraction of finding-level sentences and supports the large-scale construction of image–text pairs for Japanese VLM development.