JMIR Publications

ABSTRACT

Background:

Digitization of histopathological samples into whole slide images (WSIs) with hematoxylin and eosin (H&E) staining enables deep-learning applications for cancer detection. However, the performance of deep learning models in digital pathology is often compromised by variations in the staining protocols, scanner types, and biological differences across samples. To address this issue, a stain augmentation (SA) method was proposed, which learns the color distribution of a training dataset and generates new images using this distribution. Despite these advantages, overreliance on a single or homogeneous domain can generate similarly stained images, limiting the robustness of the model.

Objective:

In this study, we developed and validated an SA method based on stain separation that improved classification performance in H&E-stained breast cancer tissues, even when data are obtained from a single domain.

Methods:

The augmentation method was validated in both classification and segmentation tasks: 1) five domains in Camelyon17 for classification and 2) three domains in cross-organ and cross-scanner adenocarcinoma segmentation (COSAS) for segmentation, which separated the stain vector and stain density from H&E images and perturbed them by considering the counterstaining characteristics of H&E staining. The predictive model was trained for each domain and assessed in the other domains.

Results:

We obtained 348,091 and 6,989 patch images from the Camelyon17 and COSAS datasets, respectively, for image classification and segmentation tasks. Significant differences were observed in the mean color channel values across all domains in both the hue-saturation-value (HSV) and CIELAB color spaces (P < 0.05). The augmentation approach improved performance over the baseline model, yielding area under the receiver operating characteristic curve (AUROC) and area under the precision-recall curve (AUPRC) values of 0.949 (+0.142) and 0.831 (+0.306), respectively, compared to the baseline model’s values of 0.807 and 0.525. Combining this technique with RandStainNA further enhanced performance, achieving AUROC and AUPRC values of 0.955 (+0.019) and 0.857 (+0.048), compared to RandStainNA alone (0.936 and 0.808). Additionally, the feature extractor trained with this augmentation demonstrated reduced prediction error for color-jittered images compared to other methods, with an average L2 distance of 0.092 and cosine similarity of 0.992. Under substantial color variation, the mixed approach maintained robust classification performance, achieving AUROC and AUPRC values of 0.887 (+0.054) and 0.758 (+0.081), outperforming RandStainNA (0.833 and 0.677) and the augmentation approach alone (0.852 and 0.705)

Conclusions:

Stain augmentation using stain separation enhanced the performance of deep learning without data dependence in the segmentation and classification of heterogeneous H&E-stained breast cancer images.