JMIR Publications

ABSTRACT

Background:

Open Evidence (OE) is a retrieval-augmented generation (RAG) clinical AI platform. Systematic validation of citation accuracy in subspecialty knowledge bases used as educational resources is limited, especially for bilingual platforms where cross-language integrity adds complexity. The CASA (Chinese American Society of Anesthesiology) Digital Library is a bilingual (English–Chinese) anesthesia knowledge platform spanning 28 subspecialties, built primarily using OE.

Objective:

To evaluate citation accuracy of an AI-generated subspecialty anesthesia knowledge base through a completed two-stage paired validation combining a current-generation large language model (LLM; Claude Opus 4.7) with WebSearch and independent expert anesthesiologist review.

Methods:

We conducted a two-stage paired validation across 19 of 28 CASA subspecialties (570 citations; 30 per subspecialty; four-tier stratified random sampling). Each citation was independently scored by AI (Claude Opus 4.7 with WebSearch) and a blinded subspecialty expert on the four-criterion Information Retrieval (IR) framework—Hallucination, Relevance, Scientific Accuracy, Translation—using a PASS/MINOR/FAIL rubric (overall verdict PASS, CONDITIONAL, or FAIL). The prespecified primary inter-rater endpoint was a binary retain-vs-remove decision (USABLE = PASS or CONDITIONAL; NOT-USABLE = FAIL), reported with observed agreement and Gwet AC1. Three-category Cohen κ and linearly weighted κ were reported as secondary statistics.

Results:

Across 570 citations, no true hallucinations were identified (PI-canonical rate 0/570, 0%); all six AI- or human-flagged hallucination claims resolved to real publications on direct DOI/PMID verification. Stage 1 AI returned 97.0% PASS-or-CONDITIONAL (51.9% PASS, 45.1% CONDITIONAL, 3.0% FAIL); Stage 2 human experts returned 87.0% PASS-or-CONDITIONAL (74.4% PASS, 12.6% CONDITIONAL, 13.0% FAIL). On the prespecified retain-vs-remove primary endpoint, AI and humans agreed on 491/570 citations (86.1%; 95% CI 83.2–88.9%; Gwet AC1 = 0.838; almost-perfect). Exact three-category agreement was substantially lower (47.7%; weighted κ = 0.091), driven by AI's heavier use of the intermediate CONDITIONAL grade (45.1% vs 12.6% for humans), and asymmetric discordance toward AI conservatism (1.84:1).

Conclusions:

In a completed two-stage paired validation, AI and an expert anesthesiologist panel agreed on the retain-vs-remove decision 86.1% of the time (Gwet AC1 = 0.838), with 0% PI-canonical hallucination. AI defaulted to caution where humans committed — informative for triage but not equivalent to grade-level agreement. For subspecialty medical-education knowledge bases, hybrid AI-plus-expert curation — using AI for scalable initial screening and human experts for final adjudication — represents the most practical path toward reliable, bilingual, citation-verified educational resources.