JMIR Publications

ABSTRACT

Background:

Traditional rule-based natural language processing approaches in electronic health record systems are effective but time-consuming when utilizing unstructured data. These approaches require substantial efforts to parse and extract information. However, recent advancements in large language model (LLM) technology provide opportunities to automatically understand context, infer pathologic staging, and provide clear justifications for classification.

Objective:

To evaluate the applicability and performance of generative language models to automatically infer pathologic TN classifications and extract the rationale for these classifications from lung cancer surgical pathology reports.

Methods:

We fine-tuned and evaluated open-source LLMs using 3,216 lung cancer surgical pathology reports from a tertiary hospital. Considering the limited computing resources, we selected and compared six lightweight language models. Generative language models were fine-tuned to simultaneously extract and produce pathologic TN classification predictions and rationale.

Results:

Generative language models that were fine-tuned to diagnose pathologic TN stages accurately predicted TN classification and generated rationale. The highest performing model achieved an exact match ratio of 0.934 for TN classification and a semantic match ratio of 0.866 for rationale. The semantic match ratio for both predicting TN classification and presenting rationale was 0.864.

Conclusions:

This study demonstrated that fine-tuned generative language models can accurately predict TN classification and automatically extract rationale from lung cancer surgical pathology reports. These models can enhance efficiency, reduce human error, and improve accuracy in TN classification of lung cancer, thereby aiding clinical and research fields.