JMIR Publications

ABSTRACT

Background:

Structured reporting is essential for improving the clarity and accuracy of radiological information. Despite its benefits, the European Society of Radiology (ESR) notes that it is not widely adopted. This raises the need for automatic methods to extract relevant information from unstructured radiology reports and thereby create structured reports automatically.

Objective:

This study explores to combine a Bidirectional Encoder Representations from Transformers (BERT) architecture with the linguistic concept of frame semantics to extract and normalize information from free-text mammography reports.

Methods:

After creating an annotated corpus of 210 German reports for fine-tuning, we generate several BERT-model variants by applying three pre-training strategies on hospital data. Afterwards, a fact extraction pipeline is built, comprising an extractive question answering model and a sequence labelling model. We evaluate all model variants quantitatively using common evaluation metrics (model perplexity, squad_v2, seqeval) and perform qualitative evaluation of the whole pipeline by clinicians on a manually created synthetic dataset of 21 reports.

Results:

Our system is capable of extracting 14 fact types and 40 entities from the clinical findings section of mammography reports. Further pre-training on hospital data reduced model perplexity, although not having significant impact on the two downstream-tasks. We achieved averaged F1 scores of >90 % and >80 % for question answering and sequence labelling, respectively. Qualitative evaluation of the pipeline based on synthetic data shows overall precision of 96.1 % and 99.6 % for facts and entities, respectively.

Conclusions:

The proposed BERT-based framework incorporating frame semantics effectively extracts structured information from unstructured radiology reports. This system shows promise for advancing automated structured reporting in radiology, supporting improved clarity and usability of radiological data.