Accepted for/Published in: JMIR Medical Informatics
Date Submitted: Jul 30, 2021
Open Peer Review Period: Jul 30, 2021 - Sep 24, 2021
Date Accepted: Oct 20, 2021
Date Submitted to PubMed: Dec 1, 2021
(closed for review but you can still tweet)
Warning: This is an author submission that is not peer-reviewed or edited. Preprints - unless they show as "accepted" - should not be relied on to guide clinical practice or health-related behavior and should not be reported in news media as established information.
Evaluating Diagnostic Accuracy of a New Artificial-Intelligence Driven Diagnostic Support Tool
ABSTRACT
Background:
Diagnostic decision support systems (DDSSs) are computer programs aimed to improve healthcare by supporting clinicians in the process of diagnostic decision making. Previous studies demonstrated their ability to enhance clinicians’ diagnostic skills, prevent diagnostic errors, and reduce hospitalization costs. Despite their potential benefits, their utilization in clinical practice is limited, emphasizing the need for new and improved products.
Objective:
To conduct a primary evaluation of diagnostic performance for “Kahun”, a new artificial intelligence driven diagnostic tool.
Methods:
Diagnostic performance was evaluated based on the program’s ability to “solve” clinical cases from the USMLE®-step-2-clinical-skills board-exams simulations. Cases were entered to Kahun by three blinded physicians, unexperienced with the platform. The generated differential-diagnoses (DDX) were recorded and compared to the expected ones. The cases were drawn from the case-banks of three leading preparation companies: UWorld, Amboss, and FirstAid. Each case included 3“correct” differential-diagnoses. Diagnostic performance was measured in two ways. First, as sensitivity, calculated as the total number of expected DDX appropriately suggested by Kahun divided by the total number of expected diagnoses in all cases. Second, as case specific success rates, calculated as the number of cases with 1/3,2/3 and 3/3 of expected DDX appropriately suggested by Kahun divided by the total number of cases.
Results:
91 clinical cases were included in the study with 78 different chief complaints, and 174 different DDX. Kahun correctly suggested 231 diagnoses, resulting in an overall sensitivity rate of 84.9%which was stable across different disciplines. In 63.8%of the cases Kahun correctly suggested 3/3 of expected DDX within the topmost likely diagnoses, in 89%at least 2/3, and in 97.8%at least 1/3.
Conclusions:
Kahun demonstrates an acceptable diagnostic accuracy and comprehensiveness. Clinical Trial: Not applicable
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.