JMIR Publications

ABSTRACT

Background:

Novice healthcare staff often write case reports during early clinical training. However, many institutions lack structured feedback systems because of time constraints and instructor shortages. Large language models (LLMs), a form of artificial intelligence (AI), offer new opportunities to enhance educational feedback, yet their application in clinical training requires careful design to ensure pedagogically appropriate and ethically sound outputs.

Objective:

This tutorial provides a practical guide for educators without programming experience to develop an AI-based feedback system using three accessible tools: Dify (an AI workflow platform), Slack (a messaging application), and Google Apps Script. The system balances educational quality with operational efficiency while incorporating data privacy safeguards for clinical educational content.

Methods:

We developed a feedback system comprising four AI chatbots with two distinct approaches: "loop-based" bots that promote clinical reasoning through iterative, comment-based feedback, and "single-shot" bots for efficient proofreading and summarization tasks. The tutorial describes the system architecture, feedback design principles grounded in formative assessment theory, a step-by-step implementation guide, and data privacy safeguards including a de-identification protocol and API-based data protection measures. To illustrate feasibility, we conducted a pilot implementation at a community care hospital from April to June 2024, involving five novice staff members and five instructors.

Results:

A pilot implementation at a community care hospital demonstrated that the system was feasible to deploy and operate within routine clinical education workflows. Participant feedback indicated high usability and suggested that the iterative, comment-based feedback approach supported learner engagement, while also identifying areas where feedback criteria required refinement to better match institutional educational goals.

Conclusions:

This tutorial provides a reproducible framework for building a customized AI feedback system that combines comment-based iterative feedback with human-in-the-loop oversight and a multi-layered data privacy framework. By following this guide, educators can implement an adaptive system tailored to their institutional context and clinical domain.