JMIR Publications

ABSTRACT

Background:

Negation in radiology reports—for example, "no evidence of dissection"—is essential for accurately conveying diagnostic information. Clinical natural language processing (NLP) systems, however, often fail to determine whether findings occur within negated or speculative contexts unrelated to abnormal findings, leading to excessive false keyword alerts, noise in information extraction, and reduced precision in downstream clinical or research applications. Transformer‑based models have advanced detection capability but require costly expert labeling. Large language models (LLMs) possess strong linguistic understanding and may provide label‑efficient alternatives through few‑shot prompting or knowledge distillation.

Objective:

To systematically evaluate LLMs and transformer models under different supervision strategies—including comparisons between expert‑labeled and LLM‑distilled transformers, across teacher LLMs (GPT‑5, Gemma 4, and Qwen 3.6‑Plus), sampling approaches, and direct few‑shot LLM inference—to determine whether LLM inference or LLM‑based distillation can substitute for human-labeling in radiology‑report negation detection while reducing annotation costs.

Methods:

We reused and analyzed 6,000 de‑identified English‑language radiology reports from three hospital branches. Using structured prompts, three LLMs—GPT‑5, Gemma 4, and Qwen 3.6‑Plus—performed 0‑, 5‑, and 10‑shot inference to identify negations and speculations unrelated to abnormal findings and generate pseudo‑labels for transformer fine‑tuning (Bio_ClinicalBERT, BERT, and ALBERT). We trained models under expert‑labeled or LLM‑distilled supervision across 28 training data scales (42-4,200) and compared across multiple supervision and training conditions encompassing label source, teacher model, and sampling strategy. We assessed performance using keyword‑level precision, recall, and F1 with paired bootstrap tests (10,000 replicates).

Results:

We refined 1,000 labels (interrater κ = 0.957) and fine-tuned 420 transformer models. Expert‑labeled transformers consistently outperformed distill‑labeled ones, with the largest gap at small training scales (ΔF1 ≈ 0.045; p < .001) that narrowed at larger scales (ΔF1 ≈ 0.021; p < .001). Teacher model choice had minimal effect on distilled performance. ALBERT reached the highest F1 (0.990; N = 4,200) under expert supervision. In the 10‑shot setting, GPT‑5 achieved F1 = 0.976, comparable with human‑labeled transformers, and significantly outperformed Gemma 4 (0.948; p = .03) and Qwen 3.6‑Plus (0.953; p = .03) in the bootstrap analysis. GPT‑5 labeled 6,000 reports for USD 178—approximately USD 0.03 per report or USD 0.99–1.98 per hour of time‑equivalent expert labeling.

Conclusions:

Human‑supervised transformers remain the strongest baseline, but frontier LLMs such as GPT‑5 approach human-supervised performance with minimal supervision. LLM‑based distillation offers a cost‑effective alternative when annotation resources are limited, with only minor performance loss. Dataset scale proved more influential than sampling strategy. Additionally, teacher superiority yielded no proportional gains through distillation, as open‑weight or low‑cost teachers achieved comparable results. The proposed label‑efficient framework enables accurate and affordable negation detection and rapid domain adaptation.