JMIR Publications

ABSTRACT

Background:

Code-switching between Egyptian Arabic (ARZ) and English is ubiquitous in clinical settings across the Arab world, yet no dedicated benchmark exists for evaluating Automatic Speech Recognition (ASR) systems under these conditions. Existing Arabic ASR benchmarks evaluate models on Modern Standard Arabic or single-dialect speech without medical vocabulary or code-switching.

Objective:

To introduce the Clinera ASR Benchmark, the first benchmark targeting medical ARZ-EN code-switched speech, and to evaluate commercial and open-source ASR systems on accurate recognition of medical terminology using both standard and novel medical-term-aware metrics

Methods:

We curated 683 utterances of Egyptian Arabic-English medical speech with dense medical-term annotations. We evaluated six ASR systems: the proprietary ElevenLabs Scribe v1, domain-adapted ArAZN-Whisper-S (244M parameters), Whisper-Large-V3, SeamlessM4T v2, Wav2Vec2 Large AR, and Whisper-Medium. We computed standard ASR metrics (WER, CER, MER) and proposed novel medical-term metrics (MT-Precision, MT-Recall, MT-F1, MT-wF1 weighted by inverse document frequency). Statistical comparisons used bootstrap confidence intervals and Wilcoxon signed-rank tests with Bonferroni correction.

Results:

ElevenLabs Scribe v1 achieved the lowest WER (27.8%) and highest MT-wF1 (60.5%), outperforming all open models by more than six times its size on medical-term recognition. General-purpose multilingual models achieved higher WER while almost entirely failing to recognize medical terms. Among open models, ArAZN-Whisper-S dominated on medical-term efficiency (26.0 MT-wF1 per 100M parameters). Even the best commercial system produced 11 fatal or high-risk medical errors affecting 1.6% of the corpus.

Conclusions:

Current ASR systems exhibit a critical gap between general transcription accuracy and medical-term recognition in code-switched Arabic-English speech. Our benchmark and novel MT-metrics provide the first standardized evaluation framework for this clinically important setting, revealing that even top-performing systems pose patient safety risks through medical terminology errors.