Previously submitted to: JMIR Medical Education (no longer under consideration since Oct 13, 2026)
Date Submitted: Sep 1, 2026
Warning: This is an author submission that is not peer-reviewed or edited. Preprints - unless they show as "accepted" - should not be relied on to guide clinical practice or health-related behavior and should not be reported in news media as established information.
A Generative Artificial Intelligence Assistant in Medical Accreditation Examinations (GAIA-Ex)
ABSTRACT
Background:
Large language models (LLMs) have emerging applications in healthcare, including medical education and content generation. Specialist board examination preparation remains resource-intensive and relies heavily on manual question curation.
Objective:
We evaluated generative artificial intelligence (AI) as an adjunct for generating specialist board-style questions and revision material while developing a novel structured evaluation framework.
Methods:
The six-domain Veracity, Applicability, Coherence, Accuracy, Time-appropriateness and Elaboration (VACATE) framework was developed, with lower scores indicating more favourable evaluations. Examination-style questions and answers on prostate cancer were generated using Claude 3.7, Claude 4.1 and Claude 4.5. Three consultant urologists assessed clinical validity and examination suitability, while three urology residents evaluated readability and educational utility. Model versions were compared using Friedman tests with Holm correction, followed by exploratory pairwise Wilcoxon signed-rank tests.
Results:
Both learners and examiners provided generally favourable ratings across the VACATE domains. Among learners, aggregate mean scores improved from 2.11 for Claude 3.7 to 1.78 for both Claude 4.1 and Claude 4.5. Learner-rated Veracity and Applicability improved numerically across successive versions. Among examiners, Claude 4.5 achieved the most favourable aggregate mean score (2.10), followed by Claude 3.7 (2.13) and Claude 4.1 (2.65). Examiner ratings showed a non-linear pattern, with Claude 4.1 receiving less favourable Accuracy and Applicability ratings. Claude 4.5 demonstrated the most balanced descriptive profile across both groups. No VACATE domain differed significantly among versions after Holm correction, and exploratory pairwise comparisons were similarly non-significant.
Conclusions:
GAIA-Ex provides a feasible approach to AI-assisted specialist examination preparation, while VACATE enables multidimensional evaluation beyond factual accuracy. Although descriptive patterns differed between model versions, the small number of raters limited statistical power; therefore, non-significance should not be interpreted as equivalence. Larger-scale validation is required.
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.