JMIR Publications

ABSTRACT

Background:

Simulation-based learning (SBL) has become standard practice in educating healthcare professionals to apply their knowledge and skills in patient care. While SBL has demonstrated its value in education, many educators find the process of developing new, unique scenarios to be time intensive, creating limits to the variety of issues students may experience within educational settings. Generative artificial intelligence (AI) platforms, such as ChatGPT, have emerged as a potential tool for developing simulation case studies more efficiently, though little is known about the performance of AI in generating high-quality case studies for interprofessional education.

Objective:

In this study, a transdisciplinary team employed five AI platforms to generate geriatric case scenarios and systematically evaluated them for quality, accuracy, and bias.

Methods:

Ten (10) geriatric case studies were generated using the same prompt from five different generative AI platforms (N=50): ChatGPT, Claude, Co-Pilot, Gemini, and GROK. An evaluation tool was developed to collect quantitative and qualitative evaluative data to assess the quality of each case, sociodemographic data of the featured patient, the appropriateness of each case for interprofessional education, and potential bias. Case quality was evaluated using the Simulation Scenario Evaluation Tool (SSET). Each case was evaluated by three members of the research team who had experience in SBL education. Assessment scores were averaged and qualitative responses were extracted to triangulate patterns found in the quantitative data.

Results:

While each AI platform was able to generate 10 unique case studies, the quality of studies varied within and across platforms and some cases contained incomplete information. While the cases reflected diverse patient populations and contexts, some patient populations and common conditions among older adults were underrepresented or absent across the 50 case studies. All cases were set within traditional health care settings (e.g., hospital, routine medical visits). No cases featured home-based care visits. Generally, evaluators felt that the content presented in the cases was accurate, though they felt that some cases were not realistic. Evaluators also expressed that the platforms performed well at providing details and contextual information about each case, though many left out information about supplies and materials that may be available in the hypothetical scenarios.

Conclusions:

While this study demonstrated that generative AI platforms could quickly generate a diverse set of case studies that could be used for interprofessional SBL activities, educators need to review the cases they generate to make sure they provide the details needed to meet student learning objectives. This is particularly true for activities where specific patient populations, health conditions, or settings are the intended focus for learning. Educators would also benefit from training aimed at improving their prompt engineering so they can generate cases that best fit the educational needs for students.