Accepted for/Published in: JMIR Formative Research
Date Submitted: Feb 16, 2021
Date Accepted: May 31, 2021
Date Submitted to PubMed: Jun 11, 2021
Clinical utility and functionality of an artificial intelligence application to predict mortality in COVID-19: a mixed methods analysis.
ABSTRACT
Background:
Artificial neural networks (ANN) are an increasingly important tool in the context of solving complex medical classification problems. However, one of the principal challenges in leveraging AI technology in the healthcare setting has been the relative inability to translate complex models into clinician workflow at the point of care, in a time-efficient manner for end-users.
Objective:
Here we delineate the development of a COVID-19 outcome prediction application which utilises an ANN and assess its usability in the clinical setting.
Methods:
Usability assessment was conducted on the application using clinical vignettes followed by a semi-structured end-user interview. Usability was specified by effectiveness, efficiency, and satisfaction measures, reported with descriptive statistics. End-user interview data were analysed using a thematic framework, developing themes from the interview narratives.
Results:
Thirty-one Nation Health Service (NHS) physicians at a London teaching hospital, ranging from first year post-graduate through to consultants (post-graduate year 20+). All participants were able to complete the assessment, with a mean time for each patient vignettes of 59.35 seconds (standard deviation (SD) = 10.35). Mean system usability scale (SUS) score was 91.94 (SD = 8.54), which corresponds with an adjective rating of “Excellent”. Thematic analysis described positive themes around (i) the intuitive user interface, and (ii) its utility as a clinical predictive tool. A negative theme was identified around (iii) The primary concern related to use of the application in isolation as opposed to in conjunction with other clinical parameters, yet most clinicians felt that the application could positively reinforce or validate their clinical judgement.
Conclusions:
Translating AI technologies into the clinical setting remains an important but challenging task. We demonstrate the effectiveness, efficiency, and system usability of a web application designed to predict COVID-19 patient outcomes from an ANN.
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.