JMIR Publications

ABSTRACT

Background:

Disease-agnostic platforms (DAPs) are a new type of software as medical device (SaMD) that can support patients across a range of clinical conditions. Through configurability and broad intended use, DAPs aim to address the current fragmentation of digital health caused by condition-specific Digital Health Interventions (DHIs). However, this versatility presents challenges for evaluation and certification, as DAPs must be assessed across multiple therapeutic areas and diverse applications. A core challenge lies in identifying suitable evaluation frameworks that can accommodate the highly adaptable nature of this technology.

Objective:

This review aims to identify whether there are suitable evaluation frameworks which can be used for the appraisal and subsequent certification of DAPs, whether any have been applied in previous evaluations of Digital Health Interventions (DHIs), and whether any were available through open-access (OA) publications.

Methods:

Twelve databases were searched on 28 January 2024. Titles, abstracts and full texts were screened by one reviewer with a minimum 10% sample dual screened. Evaluation frameworks identified from the search were combined with frameworks identified from five previous reviews alongside frameworks identified from handsearching and grey literature. Each framework was then assessed against a set of essential and desirable criteria. Essential criteria were: the applicability across different therapeutic areas and patient populations, applicability to all domains of digital health, the ability to support both formative and summative evaluation, and availability to researchers. Frameworks which met all essential criteria were assessed against desirable criteria which included: presence of operational guidance, evidence considerations specific to digital health, and incorporation of an evidence assessment method.

Results:

From 40,907 search results, 72 new frameworks were identified. These were combined with 181 frameworks identified from previous reviews and 1 framework from handsearching and grey literature. Of the 254 frameworks assessed, 15 met all essential criteria and thus were potentially suitable for DAP evaluation. One framework, the WHO guideline on monitoring and evaluating digital health interventions, met all desirable criteria. All suitable frameworks had been previously applied to DHIs, although the number of citations at least 50-fold between frameworks. All suitable frameworks were available in at least one OA publication.

Conclusions:

This review confirm that at least 15 existing frameworks appear to be suitable for DAP evaluation and can be used to benchmark DAPs against other DHIs. That these frameworks have been used for previous evaluation of DHIs and are available in at least one OA publication, facilitating standardised, high-quality evaluation. That existing frameworks are potentially suitable indicates that research efforts can focus on answering other outstanding questions related to DAP evaluation. These include the acceptable performance benchmark for DAPs, the range of therapeutic areas DAPs need to be favourable evaluated in, and how to synthesise findings across multiple therapeutic areas. Clinical Trial: Protocol registered on OSF: https://doi.org/10.17605/OSF.IO/X578S