JMIR Publications

ABSTRACT

Background:

A breast cancer diagnosis often leaves patients with numerous questions, underscoring the need for personalized, linguistically accessible information to facilitate adherence and shared decision-making. Despite the availability of high-quality information in oncology guidelines and other materials, the challenge remains to tailor this information to the specific needs of individual patients.

Objective:

This study investigates the potential of Artificial Intelligence, specifically Large Language Models like ChatGPT, to provide personalized answers to patient inquiries in German. We present 104 real patient questions, propose evaluation criteria and compile a list of relevant documents for chatbot development.

Methods:

We collaborated with one of Germany's largest breast cancer patient representation groups to gather frequently asked questions. We tailored a custom prompt for ChatGPT and compiled a comprehensive database of pertinent medical documents, including guidelines, recommendations, and informational materials, which we then uploaded to a vector store. This database enabled ChatGPT to retrieve relevant information in response to user queries. We tested our Chatbot implementation using these questions and evaluated its answers based on four criteria: comprehensibility, correctness, completeness, and potential to cause undue harm. Each criterion was assessed on a 5-point Likert scale.

Results:

Out of 118 questions received, 104 had a medical focus, with a high information need for endocrine-based therapies and side effect management. The questions and our performance evaluation can be used as a benchmark for development of German-speaking patient-centered chatbots. The chatbot performed well overall, with 86% of answers rated as comprehensible, 88% as correct, and 89% as not harmful. However, only 69% of answers included all relevant information, and 6% were potentially harmful, underscoring the need for supervision.

Conclusions:

AI-based chatbots with access to high quality information can effectively personalize information for patients, enhancing patient-centered communication in oncology. However, supervision by medical professionals is strongly recommended.