JMIR Publications

ABSTRACT

Background:

The rapid proliferation of medical apps has transformed the healthcare landscape, providing patients and providers with unprecedented access to personalized health information and services. However, concerns regarding the apps' effectiveness and safety have raised questions about the efficacy of randomized controlled trials (RCTs) for their evaluation and as a requirement for a regulation as a mobile medical device.

Objective:

This research project addresses this issue by investigating alternative methods besides RCTs for evaluating and regulating medical apps.

Methods:

Using a qualitative approach, a focus group study with 46 international and multidisciplinary public health experts (split in three groups) was conducted at the 17th World Congress on Public Health in May 2023 in Rome (Italy) to gather in-depth insights on alternative approaches in evaluation and regulation. We conducted a policy analysis for current regulation of medical apps as mobile medical devices for the four most represented countries in the workshop: Italy, Germany, Canada, and Australia. We developed a logic model that combines the evaluation and regulation domains based on these findings.

Results:

The focus group discussions explored the strengths and limitations of current evaluation and regulation methods and identified potential alternatives that could enhance the quality and safety of medical apps. Although RCTs were only explicitly mentioned in the German regulative system (as one of many options), an analysis of chosen evaluation methods for the German apps on prescription (DiGA) pointed towards a “scientific reflex” where RCTs are always the chosen evaluation method. However, this method poses substantial limitations regarding digital interventions like medical apps. Comparable results were observed during the focus group discussions where participants expressed similar experiences for their own evaluation approaches. Additionally, the participants highlighted numerous alternatives to RCTs that can be used at different points during the life-cycle of a digital intervention to assess its efficiency and potential harm to users.

Conclusions:

It is crucial to recognize that digital interventions constantly evolve (unlike analog tools), posing challenges to inflexible evaluation methods such as RCTs. Potential risks include high dropout rates, decreased adherence, and non-significant results. Existing regulations, however, do not explicitly advocate for other evaluation methodologies. Our research highlighted the necessity to overcome the gap between regulatory demands to show safety and efficacy in medical apps and evolving scientific practices, ensuring that digital health innovation is evaluated and regulated in a way that considers the unique characteristics of mobile medical devices.