JMIR Publications

ABSTRACT

Background:

Effective communication is essential in medical practice, especially in dealing with the increasing number of patients presenting with mental health conditions. Feedback plays a crucial role in improving communication skills, but is often difficult to implement. AI-based tools offer promising support for practicing and improving communication techniques among physicians.

Objective:

This proof-of-concept study investigated the efficacy of an AI-based chatbot that provided physicians with real-time feedback to train communication skills Specifically, it addressed three research questions: (1) How accurate is the AI-generated feedback, and how is it perceived by participants? (2) Does the use of an AI-based chatbot and AI-generated feedback lead to an increased frequency in the use of specific communication techniques? (3) 3. Do physicians perceive the training with the AI-based chatbot and AI-generated feedback as beneficial for their daily clinical practice?

Methods:

The study employed a proof-of-concept design using 56 anonymized chat transcripts between physicians and simulated patients. Participating physicians received automated feedback generated by the AI-based chatbot, which was designed to assess and encourage the use of specific communication techniques. Feedback accuracy and user perception were evaluated, changes in communication behavior were assessed. Finally, participants completed a post-intervention questionnaire to evaluate perceived benefits for clinical practice.

Results:

The AI-generated feedback was found to be very accurate with 85.38% of the real-time feedback being partially or even totally correct. In addition, most of the participants considered the feedback to be accurate, with 82.14% agreeing or totally agreeing. Furthermore 85.72% of the participants agreed or totally agreed to the fact that the feedback was helpful. It also appeared to support the repeated use of recommended communication techniques. Furthermore, most of the participants agreed that practicing with an AI-based chatbot helped them practice and apply new communication techniques in clinical interactions, improved their physician-patient communication and helped them recognizing mental health conditions better in everyday medical practice.

Conclusions:

The results highlight the potential of AI-supported training to address key communication challenges in medicine, particularly in the context of mental health care. The combination of real-time practice and immediate feedback may foster sustained improvements in physician-patient interactions. Integrating such tools into medical education could offer a valuable complement to traditional training methods. Future research should look at refining AI models to improve their reliability, investigating the long-term effects and expanding demographics.