Accepted for/Published in: Journal of Medical Internet Research
Date Submitted: Jan 19, 2024
Date Accepted: Jun 21, 2024
(closed for review but you can still tweet)
Warning: This is an author submission that is not peer-reviewed or edited. Preprints - unless they show as "accepted" - should not be relied on to guide clinical practice or health-related behavior and should not be reported in news media as established information.
Comparison of Two Symptom Checkers (Ada and Symptoma) in the Emergency Department: A Randomized, Crossover, Head-to-Head, Double-Blinded Study
ABSTRACT
Background:
Emergency departments (ED) are frequently overcrowded and care for critically ill patients is delayed. Symptom checkers (SCs) promise on-demand access to reliable disease suggestions and actions to take. Contrary to the increasing use of SCs, there is a lack of supporting evidence based on direct patient use.
Objective:
To assess the diagnostic accuracy, safety, usability, and acceptance of two SCs: Ada and Symptoma.
Methods:
A randomized, crossover, head-to-head, double-blinded study including consecutive adult patients presenting to the ED at University Hospital Erlangen. Patients completed both SCs; Ada and Symptoma. The primary outcome was the diagnostic accuracy of SCs. Six blinded independent expert raters classified diagnostic concordance of SC suggestions with the final discharge diagnosis as (1) identical; (2) plausible or (3) diagnostically different. SC suggestions per patient were additionally classified as safe or potentially life-threatening and the concordance of Ada’s and physician-based triage category was assessed. Secondary outcomes were SC usability (5-Point Likert-scale; 1-very easy to use; 5-very difficult to use) and SC acceptance (Net Promoter Score).
Results:
450 patients completed the study. Most common chief complaint was chest pain (37%). The identical diagnosis was ranked first (or within top 5 diagnoses) by Ada and Symptoma in 14% (27%) and 4% (13%) of patients, respectively. An identical or plausible diagnosis was ranked first (or within top 5 diagnoses) by Ada and Symptoma in 58% (75%) and 38% (64%) of patients, respectively. Ada and Symptoma did not suggest potentially life-threatening diagnoses in 13% and 14% of patients, respectively. Ada correctly triaged, under-triaged and over-triaged 34%, 13% and 53% of patients, respectively. 88% and 78% of participants rated Ada and Symptoma as very easy or easy to use, respectively. Ada’s NPS was -34 (55% detractors; 21% promoters) and Symptoma’s NPS was -47 (63% detractors and 16%) promoters.
Conclusions:
Both SCs performed poorly compared to physicians. Ada demonstrated a higher diagnostic accuracy than Symptoma and substantially more patients would recommend Ada and assessed Ada as easy to use. The high number of unrecognized potentially life-threatening diagnoses by both SCs and inappropriate triage advice by Ada was alarming. Overall, the trustworthiness of SC recommendations appears questionable. SC authorization should necessitate rigorous clinical evaluation studies to prevent misdiagnoses, fatal triage advice, and misuse of scarce medical resources. Clinical Trial: German Register of Clinical Trials DRKS00024830; https://drks.de/search/en/trial/DRKS00024830.
Citation
