JMIR Publications

ABSTRACT

Background:

As the need for advance care planning (ACP) grows, it becomes essential to explore innovative training strategies for healthcare providers. Large language model (LLM)-based ACP chatbot may offer a promising solution to enhance healthcare providers' competence and uptake in managing intricate ACP conversations.

Objective:

To develop an ACP corpus for adapting an LLM–based ACP chatbot and to evaluate the chatbot’s performance in supporting complex ACP discussions.

Methods:

The study involved dataset construction, followed by model adaptation and evaluation. The datasets consisted of synthetic data generated using prompts derived from ACP scientific and policy texts. Both open-source and closed-source LLMs were chosen as baseline models, and adapted using fine-tuning and/or prompt engineering. Model performance was assessed through automatic and human evaluations, following the QUEST framework.

Results:

In this study, the authors created three separate datasets for the assistant, vignette, and evaluator agents, which collectively formed a multi-agent AI system for ACP training. Both automatic and human evaluations confirmed that the adapted models significantly outperformed baseline models on most aspects of Chinese ACP conversations and summarization.

Conclusions:

The multi-agent system offers an innovative, effective, and accessible approach to strengthen ACP competence, and may be integrated into ACP trainings in clinical care.