JMIR Publications

ABSTRACT

Background:

The urinary tract dilation (UTD) classification system provides objective assessment relevant to hydronephrosis management for children. However, the lack of uniform language regarding UTD in radiology reports leads to significant difficulty in both clinical management and research. Human error is also prevalent due to these inconsistencies and the heavy workload involved in reviewing and interpreting the reports.

Objective:

We seek to develop a unified model that can effectively extract UTD components and classifications from early postnatal ultrasound reports via multi-task and multi-class learning, with the potential to reduce human errors and improve the consistency and accuracy of UTD assessments.

Methods:

Radiology records from our institution were reviewed to identify infants aged 0-90 days who underwent early ultrasound for antenatal UTD. The reports and images were reviewed by the study team to establish the ground truth for UTD classification and components (primary outcome). Our multi-task multi-class model utilizes embedding layers from ClinicalBERT, a variant of the Bidirectional Encoder Representations from Transformers (BERT) model, along with 11 linear classification layers with SoftMax functions. The model's performance was evaluated using five-fold cross-validation with an 80:20 train-test split for each round. After each round, predictions that were inconsistent with the initial labeling were reviewed and corrected by human reviewers, and the model was retrained accordingly. Additionally, we compared the model's performance to GPT-3.5-Turbo in both 3-shot and 0-shot settings.

Results:

A total of 2,460 early (0-90 days) ultrasound reports were included. The five-fold cross-validated model demonstrated satisfactory performance, with a weighted F1 score greater than 0.9 for all UTD components. The model effectively identified human labeling errors, and its performance improved after correcting uncertain cases. It outperformed GPT-3.5-Turbo in all task predictions by 20%-30% in terms of accuracy.

Conclusions:

By applying deep state-of-the-art NLP neural networks, we developed a high-performing, efficient, and scalable solution to extract UTD components from unstructured ultrasound reports using one single multi-task model. This can potentially help standardize and facilitate large-scale computer vision research for pediatric hydronephrosis. Clinical Trial: N/A