Accepted for/Published in: JMIR Medical Informatics
Date Submitted: Mar 27, 2020
Date Accepted: Jun 24, 2020
Human- vs. Machine Learning Based Triage Using Digitalized Patient Histories in Primary Care
ABSTRACT
Background:
Smartphones have made it possible for patients to digitally report symptoms before physical primary care visits. Using machine learning, this data offers an opportunity to support decisions about the appropriate level of care (triage).
Objective:
To explore the inter-rater reliability between human physicians versus an automated machine learning based triage method.
Methods:
A Naive Bayes triage model was created using data from digital medical histories, capable of classifying digital medical history reports as either in need of urgent physical examination, or not in need of urgent physical examination. The classifier was tested on 300 digital medical history reports and classification was compared to the majority vote of an expert panel of five primary care physicians. Reliability between raters was measured using both Cohen’s Kappa (adjusted for chance agreement) and percentage agreement (not adjusted for chance agreement).
Results:
Inter-rater reliability as measured by Cohen's Kappa was 0.17 when comparing the majority vote of the reference group to the model. Agreement was 74% for cases judged not in need of urgent physical examination and 42% for cases judged to be in need of urgent physical examination. Between physicians within the panel, Cohen’s kappa was 0.2. Intra-rater reliability when one physician re-triaged 50 reports resulted in Cohen’s kappa of 0.55.
Conclusions:
Low inter- and intra-rater agreement in triage decisions among primary care physicians limits the possibility to use human decisions as a reference for machine learning to automate triage in primary care. Clinical Trial: Not applicable
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.