JMIR Publications

ABSTRACT

Background:

Generative artificial intelligence (GenAI) is increasingly used by medical students to summarize content While these tools may enhance efficiency, concerns have been raised that reliance on them could impede the development of summarization skills, which are essential for effective clinical communication and patient safety (de-skilling).

Objective:

To describe the development and evaluation of a GenAI-based online platform designed to train users in clinical case summarization and handoff communication through personalized feedback.

Methods:

The platform incorporates two expert-validated clinical vignettes, each with gold standard summaries for 4 target audiences (supervising physician, nurse, patient and night-shift physician). Personalized feedback is generated across three domains: content relevance, structure (based on the Situation, Background, Assessment, Recommendation [SBAR] framework), and language. Using a set of test summaries, multiple large language models (LLMs) were compared through a mixed-methods evaluation, yielding composite scores ranging from 7 (best performance) to 42 (worst performance).

Results:

Following two evaluation phases and iterative prompt optimization, Google Gemini 2.5 Pro emerged as the most reliable model, with a score of 11/42). The model consistently generated accurate, audience-appropriate feedback with few identified errors.

Conclusions:

The GenAI based platform developed in this study, powered by Google Gemini 2.5 Pro, provides high-quality feedback on clinical summaries. These findings underscore its potential as an educational tool to develop clinical communication skills. Further research should assess its effectiveness in improving learners’ summarization performance in real‑world educational settings.