JMIR Publications

ABSTRACT

Background:

Text-generating Artificial Intelligence (AI) such as ChatGPT offers many opportunities and challenges in medical education. Acquiring the practical skills necessary to use AI in a clinical context is important already during medical students’ education.

Objective:

This study aimed to investigate the feasibility of integrating ChatGPT into teaching units and to evaluate the course. Since a key domain for the use of ChatGPT could be the generation of patient information, we investigated how such information is perceived by students in terms of quality and persuasiveness.

Methods:

ChatGPT was integrated into different teaching units of a blended learning course for medical students. Using a mixed-methods approach, quantitative and qualitative data were collected. Students evaluated 4 videos featuring patient information regarding their persuasiveness on treatment expectations in a self-experience experiment and critically reviewed patient information written with ChatGPT based on different prompts. We assessed students’ characteristics, including their openness to digital innovation. The students evaluated the integration of ChatGPT into the course and their satisfaction, learning progress, and applicable knowledge based on the Kirkpatrick model and shared their thoughts regarding the future of text-generating AI in medical education.

Results:

A total of 52 medical students participated in the study. Students rated patient information generated with a basic prompt in ChatGPT as moderate in terms of comprehensibility, patient safety and the correct application of communication method. The ratings were considerably improved by the use of an extended prompt. The same text, however, showed the lowest increase in treatment expectations when compared to information provided by humans via videos (patient, clinician, expert). Findings drawn from the overall evaluation indicated high levels of satisfaction, learning progress and applicability with respect to the ChatGPT-enriched teaching units, and all the evaluation criteria were associated with each other. Higher openness to digital innovation was associated with higher satisfaction and, to a lesser extent, with higher applicability. AI-related competencies in other courses of the medical curriculum were perceived by students as highly important. Qualitative analysis highlighted potential use cases of ChatGPT in teaching and learning.

Conclusions:

The development of AI-related competencies, including the phrasing of meaningful prompts and the critical appraisal of AI-created information, is very important during medical education. This study provides valuable insights into the possible integration of these competencies into a blended-learning course to provide positive learning experiences for medical students and ensure the high perceived applicability of knowledge.