JMIR Publications

ABSTRACT

Background:

Natural language processing (NLP) is being increasing used to analyse unstructured patient feedback (UPF) for healthcare service quality improvement. Prior studies demonstrate the analytical potential of methods like topic modelling and sentiment analysis, but are largely descriptive or conceptual, with limited translation into tools for routine clinical practice. This leaves a critical knowledge gap between computational research and its realisation within (oral) healthcare service quality improvement, where large volumes of UPF are available but difficult to interpret at scale.

Objective:

Develop, implement, and evaluate a clinician-facing dashboard to translate unstructured patient feedback (UPF) into actionable insights for oral healthcare service quality improvement via a hybrid NLP Pipeline.

Methods:

We developed a four-stage hybrid NLP pipeline, combining topic modelling, sentiment analysis, and multi-label text classification. We then applied it to 57,794 reviews of NHS dental practices in England (2019–2024). Topic modelling using BERTopic identified 191 topics, with sentiment analysis via a fine-tuned DeBERTa model across four classes (positive, negative, neutral, and mixed). We iteratively consolidated topics into a ten-theme taxonomy through a hybrid approach integrating LLM-assisted classification with expert qualitative interpretation. The taxonomy informed a supervised multi-label text classifier, adapted for Google Maps Places reviews to assess portability, deploying it within a dashboard that processes real-time patient reviews, visualising thematic and sentiment insights. We evaluated the ten-theme taxonomy composition and dashboard useability through qualitative applied thematic analyses of reviews and ten semi-structured interviews with dental professionals.

Results:

Topic modelling generated 191 topics, consolidated into a ten-theme taxonomy. The sentiment classifier achieved F1=0.952 across four classes, while the multi-label theme classifier achieved micro-F1=0.765 and ROC-AUC of 0.935. Operationalised within a clinician-facing dashboard, these models enabled near real-time synthesis of patient feedback at practice level. Qualitative useability evaluation indicates the dashboard helps identify areas for service quality improvement that would otherwise be difficult to detect. Dual-axis representation of theme and sentiment enabled more nuanced interpretation, going beyond binary or single-label approaches. Overall, the dashboard helped summarise reviews for staff meetings and QI, but users tended to focus on negative feedback.

Conclusions:

Our study demonstrates a reproducible NLP pipeline to produce practical taxonomies both for oral healthcare and potentially other patient-feedback contexts. It addresses a critical knowledge gap in translating NLP research into clinician-facing tools for real-time service quality improvement. By integrating computational methods with domain-specific expert interpretation, we provide a way to bridge between data analysis and clinical application. Our findings highlight the need for hybrid approaches incorporating expert assessment to address error, bias, and useability. Meanwhile, our dashboard and mapping of its development pipeline offer a practical approach for embedding patient perspectives within routine digital healthcare to support data-driven, patient-centred service improvement in oral healthcare and beyond.