JMIR Publications

ABSTRACT

Background:

There is growing concern that artificial intelligence (AI) may diminish the quality of human relationships. However, in a context of widespread social importance (empathetic conversations between doctors and patients), we demonstrate how AI can actually improve human conversational skills potentially enhancing professional relationships. Recent advances in AI allow for realistically role-prompted counterparts for practicing professional conversations enabling relational learning without the need for human counterparts.

Objective:

The objective of our study was to show the effectiveness of AI chatbots for learning professional communicative skills in medical education. Specifically, we hypothesized that a single conversation with an AI chatbot improves communication skills in medical students across four different conversational competencies.

Methods:

We conducted a quasi-experimental intervention study involving four distinct role-prompted scenarios (ie, shared decision-making, motivational interviewing, sexually-transmitted diseases, and breaking bad news) – each designed to elicit in-depth empathic conversational skills aligned with key learning objectives in medical curricula. Students rated their competence for the four scenarios before and after a conversation with OpenAI’s GPT-4o using default settings, without fine-tuning. We expected higher competence in their conversation topic post-interaction compared to pre-interaction in a paired t-test. Participants received AI-generated feedback, which they rated regarding adequacy. Post hoc analyses addressed gender and case effects, feedback adequacy, and pre-values in competence.

Results:

Here we show that a role-prompted GPT chatbot improves self-assessed communication competence in 162 medical students after a single conversation with M=13 prompt-response-pairs (95% CI 12-14). We found an increase in communication competence of MΔ=0.94 (95% CI 0.69-1.20; d=0.58) from 5.89 [95% CI 5.55-6.23; scale 0-10] pre conversation to 6.83 [95% CI 6.55-7.12] post conversation across four different patient role prompts. Furthermore, we found participants rating AI-feedback of their conversation to be useful (M=7.92; 95% CI 7.67-8.17; scale 0-10), but feedback adequacy did not correspond to competence increase (r=.08, p=.321).

Conclusions:

Our results demonstrate how role-prompted GPT increases self-assessed communication competencies, introducing a novel tool for teaching relational learning. Our results present a starting point for using AI in education, particularly teaching communication in professional roles. Based on our findings in medical education, we anticipate further studies to investigate conversational training between lawyers and clients, marketers and customers, or managers and employees. Our research thus has implications for any field with need for conversational training and relational learning.