JMIR Publications

ABSTRACT

Background:

In the past years, accuracy of skin cancer classification by convolutional neural networks (CNNs) has improved substantially. On classification tasks of single images, CNNs have performed on par or better than dermatologists. However, in clinical practice dermatologists also use other patient data beyond the visual aspects present in a digitized image which increases their diagnostic accuracy. The effect of integration of different subtypes of patient data into CNN-based skin cancer classifiers was recently investigated in several pilot studies.

Objective:

This systematic review focuses on current research investigating the impact of merging information from image features and patient data on the performance of CNN-based skin cancer image classification. The aim is to explore the potential in this field of research by evaluating the type of patient data used, the ways the non-image data is encoded and merged with the image features as well as the impact of the integration for the classifier performance.

Methods:

Google Scholar, PubMed, Medline and ScienceDirect were screened for peer-reviewed studies published in English dealing with the integration of patient data within a CNN-based skin cancer classification. The search terms skin cancer classification, convolutional neural network(s), deep learning, lesions, melanoma, metadata, clinical information and patient data were combined.

Results:

A total of 11 publications fulfilled the inclusion criteria. All of them reported an overall improvement in different skin lesion classification tasks with patient data integration. The most commonly used patient data were age, sex and lesion location. Patient data was mostly one-hot encoded. Differences occur in the complexity that the encoded patient data was processed with regarding deep learning methods before and after fusing it with the image features for a ‘combined classifier’.

Conclusions:

The present studies indicate a potential benefit of patient data integration into CNN-based diagnostic algorithms. However, how exactly the individual patient data enhances classification performance, especially in case of multiclass classification problems, is still unclear. Moreover, a substantial fraction of patient data used by dermatologists remains to be analyzed in the context of CNN-based skin cancer classification. Further exploratory analyses in this promising field may optimize patient data integration into CNN-based skin cancer diagnostics for the benefit of the patient.