JMIR Publications

ABSTRACT

Background:

The rapid proliferation of medical apps has transformed the healthcare landscape by giving patients and providers unprecedented access to personalized health information and services. However, concerns regarding the effectiveness and safety of apps have raised questions regarding the efficacy of randomized controlled trials (RCTs) for their evaluation and as a requirement for their regulation as a mobile medical device.

Objective:

This study addresses this issue by investigating alternative methods besides RCTs for evaluating and regulating medical apps.

Methods:

Using a qualitative approach, a focus group study with 46 international and multidisciplinary public health experts, split into three groups, was conducted at the 17th World Congress on Public Health in May 2023 in Rome (Italy) to gather in-depth insights into alternative approaches to evaluation and regulation. We conducted a policy analysis of the current regulation of medical apps as mobile medical devices for the four most represented countries in the workshop: Italy, Germany, Canada, and Australia. We developed a logic model that combines the evaluation and regulation domains based on these findings.

Results:

The focus group discussions explored the strengths and limitations of current evaluation and regulation methods and identified potential alternatives that could enhance the quality and safety of medical apps. Although RCTs were only explicitly mentioned in the German regulative system as one of many options, an analysis of chosen evaluation methods for the German apps on prescription pointed towards a “scientific reflex” where RCTs are always the chosen evaluation method. However, this method poses substantial limitations regarding digital interventions such as medical apps. Comparable results were observed during the focus group discussions, where participants expressed similar experiences for their own evaluation approaches. Additionally, the participants highlighted numerous alternatives to RCTs that can be used at different points during the life cycle of a digital intervention to assess its efficacy and potential harm to users.

Conclusions:

It is crucial to recognize that digital interventions constantly evolve, unlike analog tools, posing challenges to inflexible evaluation methods, such as RCTs. Potential risks include high dropout rates, decreased adherence, and non-significant results. However, existing regulations do not explicitly advocate for other evaluation methodologies. Our research highlighted the necessity of overcoming the gap between regulatory demands to demonstrate safety and efficacy in medical apps and evolving scientific practices, ensuring that digital health innovation is evaluated and regulated in a way that considers the unique characteristics of mobile medical devices.