JMIR Publications

ABSTRACT

Background:

Generative artificial intelligence (AI) and large language models (LLMs) are increasingly used for automated item generation in health professions education, but their outputs remain vulnerable to hallucination. This vulnerability is especially consequential in pharmacy law, institutional systems, and ethics, where items must track precisely worded, periodically amended statutes. Previous studies have generated small item sets and, where evaluated, relied on exhaustive expert review, which does not scale to curriculum-wide item banks.

Objective:

This study evaluated the feasibility (completion, reproducibility, and expert agreement) of a 2-stage pipeline that generates a curriculum-scale bank of Japanese Pharmacist National Examination–style items and exhaustively screens them with an independent LLM-based legal compliance checker. It also assessed the checker's real-world performance through an AI-prioritized, human-in-the-loop expert review.

Methods:

In Stage 1, 2 Dify workflows (Gemini 2.5 Flash) generated items across a 500-topic × 3-format × 5-difficulty-level matrix (target 7500). In Stage 2, a locally run 2-round checker (Qwen2.5:32B initial judgment; Llama3.3:70B critical reverification) screened every item against statutes from Japan's e-Gov database and assigned a verdict of compliant, caution, or noncompliant. For evaluation, Stage 3 used Qwen2.5:32B to assign each item a 10-point review-priority score from the checker's output alone, and the 100 highest-priority items were selected; in Stage 4, 2 pharmacy-law faculty members independently rated them on a 0-3 usability scale (agreement assessed by Cohen κ). Post hoc audits identified a text-parsing defect in the checker's verdict-extraction code and an arbitrary file-order tie-break in the prioritization script; verdicts and the top-100 selection were recomputed from preserved outputs without new inference.

Results:

Of 7500 planned items, 7488 were valid. Under corrected verdicts, 6711 were compliant, 601 received a caution verdict, and 176 were noncompliant; the parsing defect had inflated the noncompliant count nearly 3-fold (486 vs 176) and changed 528 verdicts. Practical linked-item sets showed the highest caution rate (11.6% vs 6.3% for Essential and 6.1% for Theoretical). Of the 100 prioritized items, 40 were reclassified as compliant after correction. Because 891 items tied at a score of 8, the file-order tie-break, not the priority score, determined 53 of the 100 items, overrepresenting Essential items (51/100); a seeded random tie-break replaced 50 of the 100 items and eliminated the skew. Faculty agreement on the original set was low (unweighted κ=–0.04; linear-weighted κ=0.09), with a systematic severity difference observed between reviewers.

Conclusions:

The pipeline generated and exhaustively screened a curriculum-scale item bank, but 2 independent defects in the deterministic code—a verdict-extraction parser and a ranking tie-break—distorted the reported noncompliance rate and expert review allocation. Such code warrants the same evaluation as prompts and retrieval. The faculty review must be repeated on the corrected selection using a calibrated rubric, before operational use. Clinical Trial: None