JMIR Publications

ABSTRACT

Background:

Named Entity Recognition (NER), a key task in extracting structured information from unstructured text, such as names, dates, and locations. Traditional systems that apply NER to speech rely on a pipeline of Automatic Speech Recognition (ASR) and text-based NER, which can suffer from error propagation and performance degradation. There is a growing need to understand the capabilities of more advanced end-to-end (E2E) models that process speech directly, especially in the context of unscripted, human-to-human conversations relevant to clinical settings.

Objective:

To systematically evaluate the performance, architectures, data sources, and methodological innovations of E2E spoken NER models applied to unscripted, spontaneous human-to-human conversations.

Methods:

Following PRISMA 2020, we searched MEDLINE, Scopus, IEEE Xplore, arXiv, and other databases from inception, up to June 13, 2025. Studies were eligible if they applied E2E NER models to audio or audiovisual recordings of unscripted conversations between at least two people. Pipeline-only, text-only, scripted, and human–machine interaction studies were excluded. Data on conversational corpora, annotation schemas, model architectures, training strategies, and NER and ASR performance metrics were extracted and synthesized.

Results:

Of 1,094 records found, five studies met inclusion criteria. Four evaluated models on the SLUE-VoxPopuli benchmark, while one focused on spontaneous telephone and contact-center dialogues. All E2E approaches used self-supervised speech encoders (wav2vec 2.0, HuBERT, WavLM), and four studies employed the character-level Connectionist Temporal Classification (CTC) loss, and language-model–guided decoding. On SLUE-VoxPopuli, the strongest E2E systems achieved up to 77.2% NER F1 (ROVER ensemble) with 8.6% WER, surpassing competitive pipeline baselines. However, performance declined sharply on real-world conversational corpora, with F1 falling to 30.6% on noisy cross-domain telephone speech. E2E models required large amounts of external automatic speech recognition (ASR) and spoken language understanding (SLU) pretraining data to match or exceed pipelines, highlighting sensitivity to noise, domain shift, and limited in-domain labels.

Conclusions:

E2E models leveraging state-of-the-art self-supervised learning encoders, contextual modeling, and semi-supervised learning represent the forefront of spoken NER. While they hold significant promise, their performance on noisy, real-world conversational data highlights a critical gap that must be addressed before they can be reliably deployed in challenging domains like healthcare. Future progress will depend on the development of domain-specific datasets and models robust to the complexities of spontaneous human interaction. Registration: PROSPERO (CRD42024627286).