JMIR Publications

ABSTRACT

Background:

Classification of scanned medical reports has not been widely studied. Traditional approaches that combine Optical Character Recognition (OCR) with Machine Learning (ML) usually require preprocessing and postprocessing, yet still struggle with low-resolution images, inconsistent layouts, and multilingual content, which limits scalability. Recent advances in vision-based deep learning have demonstrated superior performance by learning discriminative features, adapting to diverse layouts, and scaling effectively with larger datasets.

Objective:

This study aims to evaluate the performance of a deep learning approach for classifying scanned medical reports.

Methods:

A vision-based deep learning pipeline is proposed for the classification of scanned medical reports. Four models are fine-tuned using transfer learning: two convolutional neural networks (EfficientNet-B7, ResNet-152) and two transformer-based models (ViT, Swin). Both partial and full fine-tuning strategies are evaluated. Model performance is assessed on training/validation splits using precision, recall, F1-score, and area under the curve (AUC). The best-performing model is further validated with 3-fold cross-validation on an independent external dataset. A dataset of 552 de-identified report images was collected from patient-contributed sources (n=242) and publicly available repositories (n=310). After preprocessing and label consolidation, reports are grouped into six categories—hematology, clinical chemistry, microbiology, serology–immunology, urinalysis, and not relevant—using a Logical Observation Identifiers Names and Codes (LOINC)-informed standardization approach. Data augmentation is applied selectively to the training dataset to address class imbalance and improve model generalization. While we do not strictly apply individual LOINC codes due to dataset limitations, the grouping is guided by LOINC’s categorical structure to ensure medical validity and consistency.

Results:

Among the four models, EfficientNet-B7 achieved the highest weighted validation F1-score (0.798), indicating comparatively balanced classification across categories, with perfect discrimination in microbiology and moderate reliability in serology–immunology (F1=0.71). ResNet-152 showed better performance in hematology (F1=0.88) and urinalysis (F1=0.93), while ViT-Huge performed best in identifying non-relevant documents (F1=0.81). On the external test set, EfficientNet-B7 yielded stable results with area under the curve (AUC) values of 1.0 for hematology, 0.88 for urinalysis, and 0.85 for non-relevant documents. Microbiology remained the most challenging category, with lower and more variable AUC values (mean 0.67).

Conclusions:

The vision-based deep learning approach achieved moderate to good performance in classifying scanned medical reports, with consistent results across internal validation and external testing. However, performance varied by category, and caution is warranted for classes with limited data, such as microbiology. These findings indicate that while the approach shows potential for supporting standardized processing of medical reports, further improvements and larger datasets are needed before routine clinical deployment.