JMIR Publications

ABSTRACT

Background:

Health professions educators are increasingly expected to design coherent teaching, clinical training, and learner assessments while balancing clinical responsibilities, limited faculty development opportunities, and local resource constraints. Instructional design (ID) provides systematic approaches for aligning learning objectives, activities, support, and assessment, whereas programmatic assessment (PA) conceptualizes assessment as a longitudinal, multisource process that supports learning and informs high-stakes judgments. However, translating these frameworks into workable local practice can be challenging.

Objective:

This study aimed to develop MedEd-Designer, a generative artificial intelligence (GenAI)-based educational design support bot integrating ID and PA, and evaluate its output characteristics, perceived usability, contextual adaptability, and risks among health professions education experts.

Methods:

We conducted an exploratory convergent mixed-methods study comprising expert-informed development and preliminary expert evaluation of MedEd-Designer. Development was informed by a structured consultation with eight multidisciplinary experts, whose inputs were translated into design principles, bot instructions, and structured reference maps comprising 20 ID and 20 PA references. Fourteen experts in health professions education and/or instructional design subsequently evaluated MedEd-Designer using two standardized vignettes (lectures and clerkships) and via their own educational activities. Pre-post ratings and free-text responses were analyzed.

Results:

In the lecture vignette, the median overall utility ratings increased from 2.0 to 4.0 for both ID and learner assessments (both P<.001). In the clerkship vignette, the corresponding ratings increased from 2.0 to 4.5 for both domains (both P<.001). Most individual items improved, particularly those related to alignment, feedback and reflection, educational theory, and the integration of multiple assessment data, whereas feasibility-related ratings showed little or no improvement. Overall bot evaluations were favorable, with median usefulness and usability ratings ranging from 4.0 to 5.0. The qualitative findings highlighted structured, theory-informed, practical, and reflective support. Limitations included contextual and resource constraints; insufficient attention to learner diversity, interaction, and psychological safety in some outputs; and the need for foundational educational knowledge and continued educator judgment.

Conclusions:

MedEd-Designer was favorably evaluated as a support tool for reviewing instructional and learner assessment design. The structured integration of ID and PA may help educators translate educational theory into practical design decisions, whereas feasibility, adaptation to local contexts, and educator oversight remain important considerations. Further real-world evaluations are required in diverse educational settings to determine whether the bot can consistently and feasibly support health professions education across diverse settings.