JMIR Publications

ABSTRACT

Background:

Lymph node metastasis (LNM) is critical for treatment decision-making of patients with resectable non-small cell lung cancer (NSCLC). But it is difficult to precisely diagnose preoperatively. Electronic medical records (EMR) contain a large volume of valuable information about LNM but some key information is recorded in free-text nature which obstacles its secondary use.

Objective:

This study aims to develop LNM prediction models based on electronic medical records using natural language processing (NLP) and machine learning algorithms.

Methods:

We first developed a multi-turn question answering NLP model to extract the features about primary tumor and lymph nodes from computed tomography (CT) reports. And then, we combined these features with other structured clinical characteristics to develop LNM prediction models using machine learning algorithms. We conducted extensive experiments to explore the effectiveness of the predictive models and compared them with size criteria based on CT image findings (the maximum short axis diameter of lymph node > 10mm were regarded as metastatic node) and clinician’s evaluation. Since the NLP model may extract features with mistakes, we also calculated the concordance correlation between the predicted probabilities of models using NLP-extracted features and gold-standard features to explore the influence of NLP-driven automatic extraction.

Results:

Experimental results show that the random forest models achieved the best performances with 0.792 area under the receiver operating characteristic curve (AUROC) value and 0.456 average precision (AP) value for pN2 LNM prediction and 0.768 AUC value and 0.524 AP value for pN1&N2 LNM prediction. And all machine learning models outperformed the size criteria and clinician’s evaluation. The concordance correlation between the random forest models using NLP-extracted features and gold-standard features is 0.950 and improved to 0.984 when top-5 important NLP-extracted features were replaced with gold-standard features.

Conclusions:

The developed LNM models can achieve competitive performance only using limited EMR data such as CT reports and tumor markers in comparison with the clinician’s evaluation. And the multi-turn question answering NLP model can extract features effectively to support the development of LNM prediction models, which may facilitate the clinical application of predictive models.