JMIR Publications

ABSTRACT

Background:

The continuously growing use of computational methods and artificial intelligence within healthcare contexts provides substantial opportunities for solutions to previously unsolvable problems. Techniques involving machine learning together with routinely collected patient data have been used to provide better tools for clinicians and improve patient outcomes. Intensive care units (ICUs) are complex and data-rich environments where critically ill patients frequently require continuous monitoring and multiple organ support. The vast amounts of data collected in the ICUs provide tremendous opportunities for machine learning, but their use comes with significant challenges. While for certain tasks the data can be used directly, more complex, and challenging problems may require additional input from humans. This input can be provided through a process of data annotation, which involves expanding on the existing data by providing information that contextualizes the data and makes it more useful for later use in the machine-learning pipeline. Annotating data is a complex, time-consuming process that requires domain expertise and frequently, technical proficiency. With the clinicians’ time being an extremely limited resource and complexities associated with the nature of healthcare data (such as multi-modality, varying formats etc.), existing data annotation tools fail to provide an effective and time-efficient solution to this problem.

Objective:

In this study, we aimed to investigate how clinicians from the ICUs approach the annotation task, focusing on the characteristics of the annotation process in the clinical setting and the potential workflow differences between different staff roles. The goal of the study was to elicit requirements for a software tool that could facilitate an effective and time-efficient data annotation.

Methods:

We conducted an experiment with clinicians from the ICUs who were asked to annotate printed data sheets with periods of time during which weaning from mechanical ventilation takes place. The participants were observed during the task and their actions were analyzed in the context of Norman’s Interaction Cycle.

Results:

The annotation process followed a loop of annotation and evaluation, during which participants incrementally analyzed and annotated the data. While no distinguishable differences were identified between how different staff roles annotate data, we observed preferences towards different methods for applying annotation which varied between different participants, as well as different admissions they were annotating. We established 11 requirements for the digital data annotation tool spread across 4 domains: annotation of individual admissions (5), semi-automated approach to annotation (3), operational constraints (2) and use of annotation in machine learning workflows (1).

Conclusions:

We conducted a manual data annotation activity to establish the requirements for a digital data annotation tool. In our analysis, we characterized the approach to the annotation exhibited by the clinicians from the ICUs and elicited 11 key requirements needed to facilitate effective data annotation in the healthcare context.