JMIR Publications

ABSTRACT

Background:

Cross-domain chest radiograph report generation is vulnerable to degradation that is not adequately captured by conventional language-quality metrics. Under dataset shift, generated reports may still look valid or broadly acceptable, yet collapse toward generic templates and preserve little case-specific clinical information.

Objective:

This study aimed to develop and apply an audit-oriented evaluation framework for cross-domain chest radiograph report generation. Special focus was placed on template-dominated degeneration, proxy clinical alignment, and the gap between apparent acceptability and practical informativeness.

Methods:

A structured audit-oriented evaluation framework was applied to an R2Gen-style multimodal report-generation backbone under bidirectional transfer between MIMIC-CXR and IU X-Ray. The framework combined training-side anti-template control, controlled decoding settings, standardized surface normalization, layered automated evaluation, and an exploratory manual audit on a fixed-seed matched subset. Automated evaluation covered formal validity, degeneration-sensitive behavior, proxy clinical alignment, and auxiliary visual-sensitivity evidence. Manual review assessed broad factual acceptability, reviewer-perceived template severity, and informational usefulness.

Results:

Across the standardized cross-domain audit pipeline, 575 evaluable outputs were analyzed for MIMIC→IU transfer and 401 for IU→MIMIC transfer. For ON-controlled settings, ON+B4 demonstrated the most conservative overall profile. In the MIMIC→IU direction, ON+B4 achieved a valid rate of 99.83%, gibberish rate of 0.17%, template rate of 0.0139, and CheX μF1 of 0.1854. In the IU→MIMIC direction, these values were 95.76%, 4.24%, 0.0175, and 0.1502, respectively. ON+G1 maintained high validity but showed higher template-related risk (template rate 0.401–0.623) and lower proxy alignment. ON+K50 exhibited the clearest overall degradation. In the exploratory manual audit (60 reports, 10 per setting-direction pair), broad acceptability ranged from 0.20–0.90, whereas informational usefulness was 0.00 in 5 of 6 cells and 0.10 in the remaining cell.

Conclusions:

No single metric fully captures cross-domain degradation in chest radiograph report generation. Outputs may remain formally valid or broadly acceptable while becoming increasingly generic and clinically uninformative. Layered auditing, combining automated and human review, is recommended for robust cross-domain evaluation of medical report-generation systems. Clinical Trial: Not applicable