JMIR Publications

ABSTRACT

Background:

Rapid growth of digital health apps has necessitated new regulatory approaches to ensure compliance with safety and effectiveness standards. Non-adherence and heterogeneous user engagement with digital health apps can lead to trial estimates that overestimate or underestimate an app’s effectiveness. Yet, there are no current standards for how researchers should measure adherence or address the risk of bias imposed by non-adherence.

Objective:

This systematic review addresses a critical question for clinical trials of software as a medical device (SaMD) apps: How and how well do researchers account for user adherence when estimating effectiveness?

Methods:

We searched FDA’s database for registrations of repeated-use patient-facing SaMDs. For each such registration, we then searched clinicaltrials.gov, company websites, and MEDLINE for corresponding clinical trials through March 2022. Adherence and engagement data was summarized for each of the 24 identified reports. Each trial report was analyzed with Cochrane Risk of Bias questions to estimate potential effects of imperfect adherence on SaMD effectiveness. This review, funded by the Richard King Mellon Foundation, is registered on Open Science Framework.

Results:

We found that trials reported adherence and engagement using a variety of definitions. For trials that reported adherence, four definitions of initiations were reported, nine definitions of implementation, and four definitions of persistence. Most trials (63%) did not report all three facets of adherence, initiation, implementation, and persistence, or use appropriate methods to evaluate efficacy (75%).

Conclusions:

Through this review, we have identified five areas of improvement for future SaMD studies: consistency of adherence reporting, reliability of adherence metrics, preregistration rates, efficacy analysis methods, and reporting of statistical methods and assumptions. We recommend ways to improve SaMD studies including by more fully reporting adherence, preregistering analyses for observational studies, and using less biased analysis methods.