JMIR Publications

ABSTRACT

Background:

Multiple-choice questions (MCQs) and single best answer (SBA) items are foundational assessment tools in health professions education, yet developing high-quality items is time-intensive and demands substantial subject-matter expertise. Following the public release of ChatGPT in November 2022, large language models (LLMs) have been increasingly applied to item generation, and the volume of related research has grown explosively between 2024 and 2026. Three new challenges have emerged alongside this growth: the unchecked circulation of unvalidated items, widening quality gaps across models and technical approaches, and the absence of an evidence-based framework to guide educators.

Objective:

This systematic review synthesizes evidence on LLM-based MCQ, SBA, and related item generation in health professions education across three practical stages: (1) generation workflow and technical feasibility, (2) expert content and quality evaluation, and (3) psychometric evaluation using learner response data. It aims to provide an evidence-based framework for the safe and effective use of AI-generated assessment items.

Methods:

Following the PRISMA 2020 guidelines, three electronic databases—PubMed, Scopus, and Web of Science—were searched from January 1, 2020, through March 24, 2026. Original empirical studies in which LLMs generated MCQ or SBA items in health professions education contexts were included; studies limited to LLM examination performance without any item generation were excluded. Three independent reviewers conducted screening and data extraction. Given substantial heterogeneity, narrative synthesis was applied

Results:

From 2,464 retrieved records, 62 studies were included, of which 60 (97%) were published in 2024 or later. GPT-4 family models were the most frequently represented model group (39/62, 63%). Evaluation consistently showed LLM-generated items to be structurally well-formed and well-aligned with learning objectives, but factual errors (0.5%–49%), poor higher-order cognitive item quality, and weak distractor plausibility were recurrently reported. Psychometric analyses indicated that LLM-generated items tended to be easier and less discriminating than human-authored items; however, domain-specific fine-tuned models and studies with structured human oversight demonstrated statistically comparable performance.

Conclusions:

LLM-based MCQ generation is technically feasible and can meaningfully reduce educator workload. However, factual errors, limited higher-order cognitive coverage, and insufficient item discrimination preclude independent AI-based item authoring for high-stakes assessments at this stage. The evidence supports a Hybrid AI-Human Workflow in which AI generates item drafts and human experts validate, refine, and monitor items using psychometric data. Immediate application is appropriate in low-stakes formative and self-study contexts, whereas high-stakes examinations require rigorous expert review and psychometric validation before deployment. Clinical Trial: no