JMIR Publications

ABSTRACT

Background:

When analyzing electronic health records, proper labeling of outcomes is mandatory. To obtain proper information from radiologic reports, several studies were conducted to classify radiologic reports using deep learning. However, the classification of pneumonia in radiologic reports written in bi-lingual languages has not been conducted previously.

Objective:

The aim of this research is to classify radiologic reports into pneumonia or no pneumonia using a deep learning method.

Methods:

For a retrospective study, our dataset included radiology reports for chest CT and chest X-rays of surgical patients from January 2008 to January 2018 in the Asan Medical Center in Korea. The classification performance of our attention-LSTM model was compared with various deep learning and machine learning methods. The AUROC (area under the receiver operating curve), AUPRC (area under the precision-recall curve), sensitivity, specificity, accuracy, and F1-score for the models were compared.

Results:

A total of 5450 radiologic reports were included which contained at least one pneumonia-related word. In the test set (n=1090), our proposed model showed 91% accuracy [AUROC for negative: 0.98, positive: 0.97, and obscure: 0.90] and the top 3 performances of the models were based on FastText or LSTM. The CNN-based model showed a lower accuracy (73%) than the other two algorithms. The classification of negative results had an F1-score of 0.96, whereas the classification of positive and uncertain results showed a lower performance (positive: 0.83, uncertain: 0.62 F1-scores). In the extra-validation set, our model showed 80% accuracy, [AUROC for negative: 0.92, positive: 0.96, obscure: 0.84]

Conclusions:

Our method showed excellent performance in classifying pneumonia in bi-lingual radiologic reports. The method could enrich the research on pneumonia by obtaining exact outcomes from electronic health data.