JMIR Publications

ABSTRACT

Background:

Electronic health records have been successfully employed in data science and machine learning projects in the past. Most of these data are collected for clinical use rather than retrospective analysis, though. This means that researchers will typically face many different issues when trying to access and prepare the data for secondary use.

Objective:

The main goal of this paper is to create awareness that preparation of routinely acquired medical data remains a challenge despite an ever-growing set of software tools.

Methods:

We report our experience and findings from a large-scale data science project analyzing routinely acquired, retrospective data from the Kepler University Hospital in Linz, Austria. The project involves data from more than 150,000 patients collected over a period of ten years. The data preparation process includes exporting the data from the hospital's data warehouses, de-identifying the data, detecting and correcting errors and inconsistencies therein, transforming them into a format suitable for machine learning, and extracting clinically meaningful labels for supervised learning.

Results:

Raw electronic health record data can be corrupted in many unexpected ways that demand thorough manual inspection and highly individualized data cleaning solutions. Specific problems encountered include: variable names or codes that differ between wards or change over time; matching data distributed across several disparate data sources; artifacts in waveform signals and challenges related to the way they are internally represented; extracting surrogate labels for supervised learning from retrospective data that lack explicit label information.

Conclusions:

Only few of the data preparation issues encountered in our project are addressed by generic medical data preprocessing tools that have been proposed recently. We propose a ‘checklist’ for guiding practitioners through retrospective medical data science projects and help them avoid the most common pitfalls. This checklist may also offer valuable insights for setting up prospective data acquisition strategies for subsequent data analysis projects.