JMIR Publications

ABSTRACT

Background:

Sepsis is a severe syndrome of organ dysfunction caused by infection, with high heterogeneity and in-hospital mortality rate, representing a grim clinical challenge for precision medicine in critical care.

Objective:

The objective of our study was to extract reported sepsis biomarkers to provide users with comprehensive biomedical information, and integrate retrieval augmented generation (RAG) and prompt engineering to enhance the accuracy, stability, and interpretability of clinical decisions recommended by large language models (LLMs).

Methods:

To address the challenge, we established and updated the first knowledge-guide clinical decision support system, MetaSepsisCDSS (http://sysbio.org.cn/sbd/), comprising 427 sepsis biomarkers and 423 researches, aiming to systematically collect and annotate sepsis biomarkers to guide personalized clinical decision-making in diagnosis and treatment of human sepsis. We curated a tailored LLM framework incorporating RAG and prompt engineering, and introduced two performance evaluation scales: System Usability Scale (SUS) and Net Promoter Score (NPS).

Results:

Surpassing baseline answers generated by GPT-4 model, the overall accuracy of expert-reviewed clinical decisions based on RAG combined with prompt engineering yielded statistically significant improvement (85.5% vs. 53.5%; p<0.001). RAG assessments (RAGAs) score between RAG-based answers and expert-provided benchmark answers illustrated prominent answer faithfulness and context relevancy than baselines. Post-use, average score of SUS was 82.20±14.17, and NPS of was 72, demonstrating high user satisfaction and loyalty.

Conclusions:

We highlight the pioneering MetaSepsisCDSS, and combining MetaSepsisCDSS with RAG and prompt engineering can minimize the limitations of precision and maximize the width in LLMs to shorten bench-to-bedside distance, serving as a knowledge-enhanced paradigm for future application of artificial intelligence in critical care medicine.