JMIR Publications

ABSTRACT

Background:

Despite the growing number of digital health (DH) applications, timely, cost-effective, and robust evaluations have not kept pace. Patient experience (PEx) was reported as one of the challenges facing the health system by the World Health Organization (WHO) in its 2018 classification of digital health initiatives report. To generate evidence of DH and promote the appropriate integration and use of technologies, standard evaluation of PEx in DH is needed.

Objective:

In this study, we systematically reviewed the literature on PEx evaluation timing considerations, evaluation indicators, and evaluation approaches in DH and generated an evaluation guide for further measurement of PEx in DH.

Methods:

We performed an umbrella systematic review following PRISMA, searching Scopus, PubMed, and Web of Science databases. Two rounds of small random sampling (20%) were independently reviewed by two reviewers who evaluated the eligibility of the articles against the selection criteria. Two-round interrater reliability was assessed using the Fleiss-Cohen coefficient (k1=0.88 and k2=0.80). Thematic analysis was then applied to analyze the extracted data based on a set of a priori categories.

Results:

The search yielded 173 records, of which 45 (26%) were eligible for analysis. Our review highlights five typical evaluation objectives which serve five stakeholder groups separately. We identified three evaluation timing considerations and classified them into three categories: intervention maturity stages, timing of the evaluation, and timing of data collection. Information on evaluation indicators of the digital PEx was identified and summarized into 3 categories (intervention outputs, patient outcomes, and healthcare system impact), 9 themes, and 22 subthemes. A set of evaluation theories, common study designs, data collection methods and instruments, and data analysis approaches were captured which can be used or adapted to evaluate the digital PEx.

Conclusions:

Our findings enabled us to generate an evaluation guide to help digital health interventions (DHI) researchers, designers, developers, and program evaluators to evaluate the digital PEx. Finally, we propose six directions for encouraging further digital PEx evaluation research and practice to address the poor PEx challenge.