JMIR Publications

ABSTRACT

Background:

The rapid advancements in natural language processing (NLP), particularly the development of large language models (LLMs), have opened new avenues for managing complex clinical text data. However, the inherent complexity and specificity of medical texts present significant challenges for the practical application of prompt engineering in diagnostic tasks.

Objective:

To address these limitations, this study proposes a novel prompt engineering strategy that integrates feature summarization, chain of thought (CoT) reasoning, and a hybrid retrieval-augmented generation (RAG) framework.

Methods:

A feature summarization approach, leveraging TF-IDF and K-means clustering, was employed to extract and distill key radiological findings. Simultaneously, the hybrid RAG framework combined dense and sparse vector representations to enhance LLMs’ comprehension of disease-related text. The proposed strategy was evaluated using a multicenter dataset containing radiology reports on pneumonia, tuberculosis, and lung cancer, with three state-of-the-art LLMs: GLM-4-plus, GLM-4-air, and GPT-4o.

Results:

Comparative analyses were performed against a BERT-based prediction model and various other prompt engineering techniques. Our strategy achieved superior performance, attaining an accuracy of 0.8947 and an F1 score of 0.8887 on the primary dataset, alongside an accuracy of 0.9167 and an F1 score of 0.8631 on an external validation dataset of radiology reports.

Conclusions:

These findings highlight the potential of LLMs to revolutionize pulmonary disease prediction, particularly in resource-constrained settings, by surpassing traditional models in both accuracy and flexibility. The proposed prompt engineering strategy not only improves predictive performance but also enhances the adaptability of LLMs in complex medical contexts, offering a promising tool for advancing disease diagnosis and clinical decision making.