JMIR Publications

ABSTRACT

Background:

The rapid growth in online symptom checkers (OSC) is not matched by advances in quality assurance. Currently there are no widely accepted criteria assessing OSCs performance. Vignette studies are widely used to evaluate OSCs, measuring accuracy of outcome. Accuracy behaves as a composite metric as it is affected by a number of individual OSC and tester dependent factors. In contrast to clinical studies, vignette studies have a small number of testers. Hence, measuring accuracy alone in vignette studies may not provide a reliable assessment of performance due to tester variability.

Objective:

(1) To investigate the impact of tester variability on accuracy of outcome of OSCs, using clinical vignettes. (2) To investigate the feasibility of measuring isolated aspects of performance.

Methods:

Healthily’s OSC was assessed using 114 vignettes, by three groups of three testers, who processed vignettes with different instructions: (a) free interpretation of vignettes (free testers), (b) specified chief complaint(s) (partially free testers) and (c) specified chief complaint(s) with strict instruction for answering additional symptoms (restricted testers). Kappa statistics were calculated to assess agreement of top outcome condition and recommended triage. Crude and adjusted accuracy was measured against a Gold standard. Adjusted accuracy was calculated using only results of consultations identical to the vignette, following a review and selection process. A feasibility study for assessing symptom comprehension of OSCs was performed using different variations of 51 chief complaints across three OSCs.

Results:

Inter-tester agreement of most likely condition and triage respectively was (a) 0.49 and 0.51 for the free tester group; b) 0.66 and 0.66 for the partially free group and (c) 0.72 and 0.71 for the restricted group For the restricted group, accuracy ranged from 43.9% to 57% for individual testers averaging 50.6%. Adjusted accuracy was 56.1%. Assessing symptom comprehension was feasible for all three OSCs. Comprehension scores ranged from 52.9% and 68%.

Conclusions:

We demonstrated that by improving standardisation of the vignette testing process, there is a significant improvement in the agreement of outcome between testers. However, significant variability remained due to uncontrollable tester dependent factors, reflected by varying outcome accuracy. Tester dependent factors combined with a small number of testers, limit the reliability and generalisability of outcome accuracy when used as a composite measure in vignette studies. Measuring and reporting different aspects of OSC performance in isolation provides a more reliable assessment of OSC performance. We developed an adjusted accuracy measure using a review and selection process to assess data/algorithm quality. Additionally, we demonstrated that symptom comprehension with different input methods can be feasibly compared. Future studies reporting accuracy need to apply vignette testing standardisation and isolated metrics.