JMIR Publications

ABSTRACT

Background:

Large language models (LLMs) are increasingly explored for clinical decision support in healthcare. However, their performance in specialized domains such as audiology remains understudied, and existing medical benchmarks may not capture the unique challenges of audiological practice.

Objective:

We introduce AUDIOLOGYBENCH, a comprehensive benchmark for evaluating LLMs in audiology along two core dimensions essential for clinical use: curated domain knowledge and literature-derived evidence assessment, complemented by detailed clinical case analysis using a standardized grading framework.

Methods:

The benchmark suite comprises three evaluation components: (i) 3,139 objective items (MCQ/TF/FIB) from educational resources, (ii) 3,175 research-article-derived items including short-answer prompts to assess research-article-based knowledge, and (iii) 67 clinically grounded case studies evaluated using a weighted scoring rubric (A-F) with domain-specific critical error classification. Four frontier LLMs (Gemini 2.5 Pro, Grok 4, OpenAI O3, and Claude Sonnet 4 Thinking) were evaluated under a unified protocol.

Results:

Clinical case analysis (primary endpoint; 804 evaluations) revealed a striking capability dissociation: recommendation generation achieved the highest scores (mean 89.74, 98.1% pass rate, zero critical errors), while audiometric interpretation showed significant limitations (mean 67.89, 35.4% critical error rate). Task type—not model selection—was the dominant performance determinant (Kruskal-Wallis χ²(2) = 268.44, P<.001, η² = 0.333 vs χ²(3) = 3.66, P=.30, η² = 0.001). On secondary endpoints, web-style MCQ exhibited ceiling effects (>95% accuracy), curated non-web items remained discriminative (35–72%), and short-answer prompts proved most challenging (best: 30%).

Conclusions:

AUDIOLOGYBENCH reveals that current LLMs demonstrate strong capability in generating clinical recommendations but exhibit significant limitations in precise numerical reporting of audiometric data. These findings suggest that LLM-assisted clinical documentation in audiology should incorporate human verification, particularly for quantitative audiometric findings.