JMIR Publications

ABSTRACT

Background:

Accelerated digitalization in the health sector requires the development of appropriate evaluation methods to ensure digital health technologies (DHTs) are safe and effective. Software as a medical device (SaMD) is a commonly used DHT by clinicians to provide care to patients. Traditional research methods for evaluating healthcare products, such as randomized clinical trials, may not be suitable for DHTs, such as SaMD. However, evidence to show their safety and efficacy is needed by regulators before they can be used in practice. Clinical simulation can be used by researchers to test SaMD in an agile and low-cost way, yet there is limited research on criteria to assess the robustness of simulations and subsequently, their relevance for a regulatory decision.

Objective:

The objective of this study was to gain consensus on the criteria that should be used to assess clinical simulation from a regulatory perspective when it is used to generate evidence for SaMD.

Methods:

An eDelphi study approach was chosen to develop a set of criteria to assess clinical simulation when used to evaluate SaMD. Participants were recruited through purposive and snowball sampling based on their experience and knowledge in relevant sectors. They were guided through an initial scoping questionnaire with key themes identified from the literature to obtain a comprehensive list of criteria. Participants voted upon these criteria in two Delphi rounds, with criteria being excluded if consensus was not met. Participants were invited to add qualitative comments during rounds and qualitative analysis was performed on the comments gathered during the first round. Consensus was pre-defined by two criteria: if <10% of the panelists deemed the criteria as ‘not important’ or ‘not important at all’ and >60% ‘important’ or ‘very important’.

Results:

33 international experts in the digital health field, including academics, regulators, policy makers, and industry representatives, completed both Delphi rounds. 43 criteria gained consensus from the participants. The research team grouped these criteria into seven domains - background and context, overall study design, study population, delivery of the simulation, fidelity, software and artificial intelligence (AI) and study analysis. These seven domains were formulated into the Simulation for Regulation of SaMD (SIROS) framework. There were key areas of concern identified by participants regarding the framework criteria, such as the importance of how simulation fidelity is achieved and reported and the avoidance of bias throughout all stages.

Conclusions:

This study proposes the SIROS framework, developed through an eDelphi consensus process, to evaluate clinical simulation when used to assess SaMD. Future research should prioritise the development of safe and effective SaMD, while implementing and refining the framework criteria to adapt to new challenges. Clinical Trial: N/A