JMIR Publications

ABSTRACT

Background:

The rapid advancements in natural language processing (NLP), particularly the development of large language models (LLMs), have opened new avenues for managing complex clinical text data. However, the inherent complexity and specificity of medical texts present significant challenges for the practical application of prompt engineering in diagnostic tasks.

Objective:

In this study, LLMs with new prompt engineering technology are explored to enhance model interpretability and improve the prediction performance of pulmonary disease based on traditional deep learning model.

Methods:

A retrospective dataset including 2965 chest CT radiology reports was constructed. The reports were from four cohorts, healthy individuals, patients with pulmonary tuberculosis, lung cancer, and pneumonia. Then a novel prompt engineering strategy that integrates feature summarization(F-Sum), chain of thought (CoT) reasoning, and a hybrid retrieval-augmented generation (RAG) framework was proposed. A feature summarization approach, leveraging TF-IDF and K-means clustering, was employed to extract and distill key radiological findings related to three diseases. Simultaneously, the hybrid RAG framework combined dense and sparse vector representations to enhance LLMs’ comprehension of disease-related text. Three state-of-the-art LLMs, GLM-4-Plus, GLM-4-air, and GPT-4o, were integrated with the prompt strategy to evaluate the efficiency in recognizing pneumonia, tuberculosis, and lung cancer. The traditional deep learning model, BERT, was also compared to assess the superiority of LLMs. Finally, the proposed method was tested on an external validation dataset consisted of 343 chest CT report from another hospital.

Results:

Comparing with BERT-based prediction model and various other prompt engineering techniques, our method with GLM-4-Plus achieved the best performance on test dataset, attaining an F1 score of 0.8887 and accuracy of 0.8947. On external validation dataset, F1 score (0.8631) and accuracy (0.9167) of the proposed method with GPT-4o were the highest. Compared to the popular strategy with manually selected typical samples (Few-shot) and CoT designed by doctor (F1:0.8294, Accuracy: 0.8342), the proposed method that summarized disease characteristics (F-Sum) based on LLM and automatically generated COT performed better (F1: 0.8870, Accuracy: 0.8974). Although the BERT-based model got similar results on test dataset (F1: 0.8514, Accuracy: 0.8763), its predictive performance significantly decreases on the external validation set (F1: 0.4836, Accuracy: 0.7833).

Conclusions:

These findings highlight the potential of LLMs to revolutionize pulmonary disease prediction, particularly in resource-constrained settings, by surpassing traditional models in both accuracy and flexibility. The proposed prompt engineering strategy not only improves predictive performance but also enhances the adaptability of LLMs in complex medical contexts, offering a promising tool for advancing disease diagnosis and clinical decision making.