Previously submitted to: JMIR Medical Informatics (no longer under consideration since Oct 18, 2019)
Date Submitted: Sep 3, 2019
Warning: This is an author submission that is not peer-reviewed or edited. Preprints - unless they show as "accepted" - should not be relied on to guide clinical practice or health-related behavior and should not be reported in news media as established information.
Medical Analytics in the Presence of Human-Error. An Exploration of EMR Data Quality using MIMIC-III.
ABSTRACT
Background:
Public Electronic Medical Records (EMR) datasets are a goldmine for vendors and researchers seeking to develop analytics designed to assist caregivers in monitoring, diagnosis, and treatment of patients. Both complex machine-learning-based tools, which require copious amounts of data to train, and a simple trend graph presented in a patient-centered dashboard, are sensitive to noise.
Objective:
We aim to systematically explore data errors in MIMIC-III as a representative of secondary use datasets and the impact of these errors on downstream analytics.
Methods:
We discuss the unique challenge of accounting for the specific patient's medical condition and personal characteristics such as age, weight, gender, and others, in identifying data errors when only a few measurements of each patient are available. To do so, we examine the prevalence and manifestations of errors in one of the most popular public medical research databases - MIMIC-III. We then evaluate how these errors impact visual analytics, score-based sepsis analytics SOFA and qSOFA, and a machine-learning-based sepsis predictor.
Results:
We find a variety of error patterns in MIMIC-III and highlight effective methods to find them. All analytics are found to be sensitive to sporadic error. Visual analytics are severely impacted, limiting their usefulness in the presence of error. qSOFA and SOFA suffer a score change of +1 (of 3) and +2.3-4 (of 15). The sepsis predictor suffers from a 0.01-0.3 score change compared to a median score of 0.08.
Conclusions:
The use of statistical methods to detect data errors is limited to high-throughput scenarios and large data aggregations. There is a dearth of medical guidelines and error-detection practices to support rule-based systems, required to keep analytics safe and trustworthy in low-volume scenarios. Analytics developers should test their software’s sensitivity to error on public datasets. The medical informatics community should improve support for medical data-quality endeavors by creating guidelines for plausible values and analytics robustness to error and collecting real-world dirty datasets which contain errors as they appear in normal EMR use.
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.