JMIR Publications

ABSTRACT

Background:

Chest computed tomography (CT) is crucial for the detection of lung cancer, and many automated CT evaluation methods have been proposed. Due to the divergent software dependencies of the reported approaches, they are rarely compared or reproduced.

Objective:

To generate reproducible machine learning modules for lung cancer detection and to compare the approaches and performance of the award-winning algorithms developed in the 2017 Kaggle Data Science Bowl.

Methods:

We obtained the source codes of all award-winning solutions of the Kaggle Data Science Bowl 2017 Challenge, where participants developed automated CT evaluation methods to detect lung cancer (training set n=1,397, public test set n=198, private test set n=506). The performance of the algorithms was evaluated by the log loss function, and Spearman’s correlation coefficient of the performance in the public and private test sets was computed.

Results:

Most solutions implemented distinct image pre-processing, segmentation, and classification modules. Variants of U-Net, VGGNet, and ResNet were commonly used in nodule segmentation, and transfer learning was used in most of the classification algorithms. Substantial performance variations in the public and private test sets were observed (Spearman’s correlation coefficient = 0.394 among the top 10 teams). To ensure the reproducibility of results, we generated a Docker container for each of the top solutions at http://rebrand.ly/chestct.

Conclusions:

We compared the award-winning algorithms for lung cancer detection and generated reproducible Docker images for the top solutions. Although convolutional neural networks achieved decent accuracy, there is plenty of room for improvement regarding model generalizability.