JMIR Publications

ABSTRACT

Background:

In silico evaluation (ISE) methods create a digital twin or a computer simulation of actual care pathways, enabling a broader assessment of the potential impact of algorithm-based clinical decision support systems (CDSS) prior to implementation.

Objective:

This study aims to review the scope of proposed ISE methods to evaluate CDSS. Specifically, it focuses on simulation modeling methods, paradigms, parameters, and outcomes considered within the ISE methodological domain.

Methods:

The current review follows widely accepted scoping review guidelines. We conceptualized our search framework and conducted a two-stage screening process of studies collected using an automated search of selected databases. Subsequently, information about CDSS and simulation modeling methods was extracted.

Results:

Our review revealed that a small subset of studies on CDSS development conducted validation studies that leveraged clinical workflow simulations. Existing studies frequently prioritize patient, process, and cost-effectiveness outcomes, while the outcomes directly related to the care providers’ well-being are lacking. Studies reviewed demonstrated a diverse collection of simulation modeling paradigms. These included discrete event simulation, agent-based modeling, and Markov modeling. Three distinct ISE objectives surfaced from the reviewed literature: (1) outcome comparison; (2) outcome comparison with sensitivity analysis, and (3) simulation-based optimization. The first two implement a decoupled CDSS model training followed by simulation-based evaluation of the trained CDSS model. The third approach simultaneously optimizes the CDSS model decision support capabilities through the simulation process, where training and evaluation take place simultaneously.

Conclusions:

The growing body of research in the development of algorithm-based CDSS calls for a shift in how evidence for their safety and reliability can be ascertained. Traditional trial-based evaluations may not align with the risk considerations and resource impact of novel algorithm-based CDSS without a compelling understanding of systems-level implications in clinical workflows. Our review examined the potential and gaps of ISE, particularly through simulation modeling, which can be applied as a pre-implementation strategy to reinforce evidence for safe and effective implementation of algorithm-based CDSS.