JMIR Publications

ABSTRACT

During skin cancer screening consultations, dermatologists are confronted with a large number of skin lesions to be evaluated. This challenging task requires expertise and involves lots of responsibility. Therefore, an automated, reliable tool that supports the diagnosis of the clinician would be an important help. Content-Based Image Retrieval (CBIR) algorithms using deep convolutional neural networks (CNNs) have been already shown to retrieve similar dermoscopic images from a given dataset based on extracted features with remarkable performance. In skin lesion imaging the relevant information is usually spatially restricted. Therefore, traditional CNN-based CBIR algorithms were expanded with an additional fine-tuning training adding saliency maps to the original image before the retrieval. We refer to this architecture as Saliency-Enhanced Content-Based Image Retrieval (SE-CBIR). A significant improvement in the quantitative retrieval of our SE-CBIR model compared to the single CNN classifier architecture was observed. To assess the relevancy of the retrieved images, a reader study was proposed to dermatologists and residents of the University Hospital Zurich. To facilitate participation in the study an online survey was developed. The outcome of the study showed an increase in classification accuracy of 22% when the participant is provided with SE-CBIR retrieved images. In addition, the overall confidence in the lesion’s diagnosis increased by 24%. Finally, the use of SE-CBIR as a support tool helped the participants to reduce the number of non-melanoma lesions previously diagnosed as melanoma (overdiagnosis) by 53%.