Accepted for/Published in: Journal of Medical Internet Research
Date Submitted: Nov 20, 2020
Date Accepted: Apr 30, 2021
Evaluation framework for the success of AI-enabled Clinical Decision Support Systems: A Mixed Design Study
ABSTRACT
Background:
Clinical decision support systems (CDSS) are designed to utilize medical data, knowledge and analysis engines, and to generate patient-specific assessments or recommendations to health professionals in order to assist decision-making. AI-enabled CDSS (AI+CDSS) aid the decision-making process through an intelligent component. Well-defined evaluation methods are essential to ensure the seamless integration and contribution of AI+CDSS to clinical practice.
Objective:
To examine a proposed evaluation AI+CDSS framework by: 1) developing and validating a measurement instrument (MI), and 2) testing the inter-relations of evaluation variables.
Methods:
An AI+CDSS evaluation framework consisting of 6 variables was developed. A Delphi process was conducted to develop the MI items. Cognitive interviews and pre-test were performed to refine the questions. Collected response data of web-based survey was analyzed to remove irrelevant questions from the MI, to test the dimensional structure, and to assess reliability and validity. The interrelations of those variables were tested and verified using path analysis.
Results:
A 28-item MI was developed. MI survey responses were collected from 156 respondents. The Cronbach’s alpha of MI was 0.963, content validity was 0.943. Values of average variance extracted ranged from .582 to .756, and values of Heterotrait-Monotrait ratio ranged from .376 to .896. Variables in the final model accounted for 89.00% of the variance in the “User acceptance”.
Conclusions:
User acceptance is the central dimension of AI+CDSS success. Acceptance was directly influenced by perceived ease of use, information quality, service quality, and perceived benefit. Acceptance was also indirectly influenced by system quality and information quality through perceived ease of use. User acceptance and perceived benefit were interrelated.
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.