JMIR Publications

ABSTRACT

Background:

Clinical decision support (CDS) algorithms and systems improve decision-making efficiency in the clinical setting. The integration of artificial intelligence enhances the performance of CDS, enabling clinical decisions to be made using extensive patient and clinical workflow information. However, the high trust and reliability requirements of healthcare applications create a gap between developing and deploying CDS systems. Addressing the gap calls for innovations in the evidence-generation process, specifically in addressing the trade-off between the scope and feasibility of traditional clinical impact assessment procedures.

Objective:

This study aims to review the scope of proposed in silico evaluation methods. Specifically, it focuses on simulation modeling paradigms, parameters, and outcomes considered in the included studies.

Methods:

The current scoping review follows procedures proposed by well-accepted review guidelines. Specifically, we conceptualized our search framework and conducted a two-stage screening process of articles collected using an automated search of selected databases. Subsequently, information about clinical decision-making domains and simulation modeling methods was extracted.

Results:

Our review revealed that a small subset of articles on CDS systems conducted validation using clinical workflow simulations. These studies frequently prioritized patient, process, and cost-effectiveness outcomes. While other studies include the simulation of care provider-related parameters such as caring capacity and adherence to CDS recommendations, evaluation of outcomes on care provider well-being is lacking. Three primary approaches to in silico evaluation were identified. We refer to them as outcome measurement, outcome measurement with sensitivity analysis, and simulation-based optimization. The first two implement a decoupled strategy through an initial CDS model training and optimization and a subsequent simulation-based evaluation. The third approach dynamically optimizes CDS model decision-making through simulation modeling—an integrated strategy. The studies under review used diverse modeling paradigms, including discrete event simulation, agent-based modeling, and Markov modeling.

Conclusions:

In silico evaluation presents significant advantages for CDS system evaluation by enabling preimplementation impact assessment, ongoing performance monitoring, and resilience testing under extreme conditions without disrupting existing healthcare infrastructures. Despite numerous studies on CDS development, in silico evaluation of such remains underutilized. When used, evaluations often focus on patient outcomes, process efficiency, and cost-effectiveness; notably, outcomes related to provider well-being are not studied. We further highlight the decoupled and integrated evaluation strategies, implementing various simulation modelling paradigms in the digital twinning of care pathways. Future work on in silico evaluation of CDS will benefit from standardized reporting to ensure consistent and transparent method application, which will enhance collaboration and knowledge sharing and contribute to achieving the quadruple aims of healthcare.