JMIR Publications

ABSTRACT

Background:

Neonatal mortality remains a leading contributor to under-five deaths globally, particularly in low- and middle-income countries (LMICs). While machine learning (ML)–based risk prediction models show promise for identifying high-risk neonates, there is limited evidence on the real-world usability and practical feasibility of implementing such predictors within routine clinical workflows in LMIC neonatal units.

Objective:

This study aimed to evaluate the usability, user experience (UX), and perceived clinical utility of a paper-based neonatal risk predictor tool derived from an ML model and implemented across three Kenyan health facilities.

Methods:

A post-implementation, cross-sectional usability evaluation was conducted following a four-month implementation period. The study was embedded within a longitudinal mixed-methods project. Frontline neonatal healthcare workers (n = 10) completed standardized usability instruments, including adapted global usability items (USQ), selected Questionnaire for User Interaction Satisfaction (QUIS) domains, and the Post-Study System Usability Questionnaire (PSSUQ), alongside a project-specific Post-Study Neonatal Utility Questionnaire (PSNUQ). Descriptive statistics (medians, interquartile ranges, and category percentages) were computed. Three purposively selected neonatal unit leaders participated in semi-structured key-informant interviews (KIIs), which were analyzed using thematic analysis. Quantitative and qualitative findings were triangulated to contextualize perceptions of usability.

Results:

Overall usability perceptions were favorable. Seventy-five percent of respondents reported willingness to use the tool frequently, and 55% indicated confidence in independent use. Fifty-two percent disagreed that the tool was complex, while 22% agreed, indicating moderate polarization in perceived complexity. Median PSSUQ composite scores were below 3 across subscales, reflecting positive usability ratings. Eight respondents (80%) agreed that the tool supports early identification of high-risk neonates and improves care prioritization within the first 48 hours. However, workflow integration was reported to be workload-sensitive: 40% reported an increased documentation burden during high patient volumes. KIIs identified staffing shortages, parallel documentation systems, and the importance of administrative endorsement as key structural influences on adoption.

Conclusions:

The neonatal risk predictor tool demonstrated acceptable usability, learnability, and perceived clinical relevance across three diverse Kenyan facilities. However, variability in perceived complexity and workload sensitivity highlights the importance of structured onboarding, workflow-aligned integration, and context-aware implementation planning. These findings underscore that successful translation of ML-derived predictor models into clinical practice requires not only technical validity but also strong usability and system-level readiness within routine neonatal care settings.