JMIR Publications

ABSTRACT

Background:

User involvement is increasingly being acknowledged as a central part of healthcare innovation. Yet, meaningful user involvement during the development and testing of digital health technologies (DHTs) is often not fully realised.

Objective:

The current study aimed to examine in which areas user input is most prevalent, and whether there is an association between user inclusion and compliance with recognised best practices for DHTs.

Methods:

A secondary analysis was conducted on an assessment dataset of 1595 DHTs collected by the Organisation for the Review of Care and Health Applications (ORCHA) between January 2021 and January 2022. The dataset contained information on whether the DHTs had been developed or tested with user input and whether they followed best practices across several domains. Additionally, background information was available regarding the DHTs’ country of origin, targeted disease areas, subjective user ratings, download numbers, and complexity/risk (as per the National Institute for Health and Care Excellence Evidence Standard Framework, NICE ESF). Descriptive statistics, Mann-Whitney tests, and Pearson chi-square analyses were applied to the data.

Results:

Overall, user input was reported by 139 (8.7%) DHTs for only the development phase, by 537 (33.7%) DHTs for only the testing phase, by 349 (21.9%) DHTs for both phases, and by 570 (35.7%) DHTs for neither phase. The highest percentage of DHTs with reported user input during development were observed in Denmark (79.2%); in the disease areas of diabetes (48.1%), cardiology (46.9%), pain management (46.5%) and oncology (46.3%); and for high DHT complexity/risk (ESF tier 3a, 39.9%). The highest percentage of DHTs with reported user input during testing were observed in Belgium (90.9%), Sweden (85.3%), and France (81.3%); in the disease areas of neurodiversity (80.8%), respiratory diseases (76.3%), cardiology (71.9%) and diabetes (70.9%); and for high DHT complexity/risk (ESF tiers 3a, 66.9%, and 3b, 64.8%). Moreover, DHTs that reported seeking user input during testing demonstrated significantly more downloads than those that did not (P=.008). No significant group differences were observed for user ratings. Furthermore, user inclusion was associated with improved compliance with best practices for clinical assurance, data privacy, risk management, and user experience.

Conclusions:

Interestingly, the countries and disease areas in which the highest percentage of DHTs with user input were observed tended to be those with higher digital maturity in healthcare and more funding availability, respectively. This suggests that there may be a trade-off between developers’ willingness/ability to involve users and the need to meet challenges arising from infrastructure limitations and financial constraints. Moreover, the finding of a positive association between user inclusion and compliance with best practices indicates that, where no other guidance is available, users may benefit from prioritising DHTs developed with user input, as the latter may be a proxy for broader DHT quality.