JMIR Publications

ABSTRACT

Background:

Medical AI education requires accurate answers. However, annotated errors can provide insight into how to teach. Therefore, inter-rater variability can cause biases in common error examples.

Objective:

To understand the extent and reliability of error categories annotated in published GPT-4 errors, and translate this into an educational competency framework and traceable teaching resources.

Methods:

It was conducted a secondary analysis of the open-source GPT-4 USMLE error dataset. This consisted of 300 answers from a publicly available set of 919 answers categorized as incorrect. Categories were inferred for 892 rating records from 45 public rater identifiers and any-rater and two-rater agreement calculated. Nominal Krippendorff alpha and two forms of set agreement were calculated. Item-bootstrap confidence intervals were used. Fisher exact tests with Holm correction identified any associations between repeated answers. Competencies and teaching cases were created from the inherited taxonomy with no new model generation or participant recruitment.

Results:

A total of 284/300 answers received at least one error category (94.7%). 122/300 answers had two or more raters providing the same error category (40.7%). The most common categories were commitment to incorrect diagnosis/answer (193 vs 65), incorrect/vague conclusion (168 vs 45) and unsupported medical statement (103 vs 19). The mean Jaccard agreement for assigned categories, including the two non-error categories, across 299 items with multiple raters was 0.226 (95% CI 0.195 to 0.258). Agreement increased to 0.662 with common negative categories (95% CI 0.647 to 0.678). Category specific alpha values ranged from -0.027 to 0.107. No associations remained after multiplicity correction from the seven associations between repeated answers and mismatch. Eight competencies, seven observed response teaching cases and an annotation literacy exercise were generated.

Conclusions:

Medical AI error labels can support educational resource development. Nonetheless, their instructional interpretation depends on annotation support and reliability. The proposed framework links observed responses to eight appraisal competencies, including evaluation of the judgments used to classify errors. Its contribution is a reproducible basis for teaching evidence verification and justified uncertainty; competency validity and educational effectiveness require independent evaluation.