Accepted for/Published in: JMIR Formative Research
Date Submitted: Oct 26, 2023
Open Peer Review Period: Oct 26, 2023 - Nov 13, 2023
Date Accepted: Apr 24, 2024
(closed for review but you can still tweet)
Longitudinal changes in diagnostic accuracy of a differential diagnosis list developed by an artificial intelligence-based symptom checker: a retrospective observational study
ABSTRACT
Background:
Artificial intelligence (AI) symptom checker models should be trained using real-world patient data to improve their diagnostic accuracy. Given that AI-based symptom checkers are currently employed in clinical practice, their performance should improve over time. However, longitudinal evaluations of the diagnostic accuracy of these symptom checkers are limited.
Objective:
This study aimed to assess the longitudinal changes in the accuracy of differential diagnosis lists created by an AI-based symptom checker employed in the real world.
Methods:
This was a single-center, retrospective, observational study. Patients who visited an outpatient clinic without an appointment between May 1, 2019 and April 30, 2022 and who were admitted to a community hospital in Japan within 30 days of their index visit were considered eligible. We only included patients who underwent an AI-based symptom checkup at the index visit, and the diagnosis was finally confirmed during follow-up. Final diagnoses were categorized as common or uncommon, and all cases were categorized as typical or atypical. The primary outcome measure was the accuracy of the differential diagnosis list created by the AI-based symptom checker, defined as the final diagnosis in a list of 10 differential diagnoses created by the symptom checker. To assess the change in the symptom checker’s diagnostic accuracy over 3 years, we used a chi-squared test to compare the primary outcome over three periods: from May 1, 2019 to April 30, 2020 (first year); from May 1, 2020 to April 30, 2021 (second year); and from May 1, 2021 to April 30, 2022 (third year).
Results:
A total of 381 patients were included. Common diseases comprised 257 cases (68%), and typical presentations were observed in 298 cases (78 %). Overall, the accuracy of the differential diagnosis list created by the AI-based symptom checker was 172/381 (45%), which did not differ across the 3 years (first year, 44%; second year, 44%; and third year, 48%; P=.85). The accuracy of the differential diagnosis list created by the symptom checker was low in those with uncommon diseases (24%) and atypical presentations (15%). In the multivariate logistic regression model, disease commonality and presentation typicality were significantly associated with the accuracy of the differential diagnosis list created by the symptom checker.
Conclusions:
A 3-year longitudinal survey of the diagnostic accuracy of differential diagnosis lists developed by an AI-based symptom checker, which has been implemented in real-world clinical practice settings, showed no improvement over time. Uncommon diseases and atypical presentations were independently associated with a lower diagnostic accuracy. In the future, symptom checkers should be trained to recognize uncommon conditions. Clinical Trial: Not applicable
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.