JMIR Publications

ABSTRACT

Background:

Evaluating artificial intelligence (AI)-generated citations requires checking article existence, clinical relevance, scientific accuracy, and translation fidelity. A single agreement score cannot capture these distinct judgments, their stability over time, or how they were reached. Repeat assessments and retained evaluation records can support transparent, inspectable citation review in bilingual educational resources.

Objective:

We quantified paired AI-human agreement at binary, three-category, and criterion levels. To inform specialist citation review, we counted citations flagged by different AI alert rules and human FAIL ratings left unflagged by those rules. We compared original and repeat AI ratings and examined each citation’s recorded evaluation steps.

Methods:

This cross-sectional study systematically sampled 570 OpenEvidence-generated citations across 19 subspecialties of a bilingual anesthesiology knowledge base. Each received one AI and one human overall verdict; four domains assessed existence, relevance, scientific accuracy, and translation fidelity. Ten validators represented nine institutions; validator and subspecialty effects were inseparable. The post hoc primary binary comparison combined PASS and CONDITIONAL. Human verdicts were a reference, not truth. Records documented search and recovery. Applicable 95% confidence intervals (CIs) used tier-preserving subspecialty-cluster bootstrapping (B=10,000). AI reassessed 112 citations to compare repeat ratings with the original ratings.

Results:

Binary agreement was 489/570 (85.79%; 95% CI 79.12%-91.75%); Cohen kappa was 0.0645 (−0.0103 to 0.1639), Gwet's first-order agreement coefficient (AC1) 0.8334 (0.7388-0.9084), and Brennan-Prediger agreement 0.7158 (0.5825-0.8351). Three-category agreement was 271/570 (47.54%; 41.93%-53.16%). Agreement was lower for relevance (323/569, 56.77%; 51.75%-61.62%) and scientific accuracy (440/568, 77.46%; 72.63%-81.83%). Flagging AI CONDITIONAL/FAIL identified 274/570 citations (48.07%; 42.98%-52.98%) and captured 46/74 human FAIL (62.16%; 48.27%-74.42%); positive predictive value was 46/274 (16.79%; 7.89%-26.27%). Among unflagged citations, 28/296 were human FAIL (9.46%; 4.84%-14.81%). In a later AI and retrieval run, unweighted agreement with original AI ratings was 91/112 (81.25%) for three categories and 107/112 (95.54%) for two categories. Original AI FAIL ratings persisted for 12/17 overall and 2/4 for citation existence. From human records, 389 citations were assigned no issue by rule without individual rereview; the principal investigator individually reviewed 181. Post hoc classification assigned 154/570 (27.02%) to primary issue categories 1–5, including 33/154 verification gaps; this is not error prevalence.

Conclusions:

AI-generated citations need evaluation as bibliographic references and evidence supporting clinical statements. This study combines paired AI-human judgments of the same citations, specific citation checks, repeat AI assessment, and further human review within a working bilingual knowledge base. The framework shows how evaluation choices shape the interpretation of agreement and highlights concerns for specialist review. Specialists can check whether source passages support claims and whether summaries and translations preserve scientific meaning. Retained records support examination of judgments, while public structured ratings enable statistical recalculation and reuse. These resources contribute to metascience and research integrity by making citation evaluation inspectable as AI-assisted evidence use expands.