JMIR Publications

ABSTRACT

Background:

Limited data accuracy is often cited as a reason for caution in the integration of physiological data obtained from consumer-oriented wearable devices in care management pathways. The effect of decreasing accuracy on predictive models generated from these data has not previously been investigated.

Objective:

The objective of this study is to simulate the effect of data degradation on the reliability of prediction models generated from that data and thus determine the extent to which lower device accuracy might or might not limit their use in clinical settings.

Methods:

Using the Multilevel Monitoring of Activity and Sleep in Healthy People (MMASH) dataset, which includes continuous free-living step count and heart rate data from 21 healthy volunteers, we trained a random forest model to predict cardiac competence. Model performance in 75 perturbed datasets with increasing missingness, noisiness, bias and a combination of all three perturbations was compared to model performance for the unperturbed dataset.

Results:

The unperturbed dataset achieved a mean root mean square error (RMSE) of 0.079±0.001 in predicting cardiac competence index. For all types of perturbations, RMSE remained stable up to 20-30% perturbation. Above this level, RMSE climbed to a point at which the model was no longer predictive at 80% for noise, 50% for missingness and 35% for the combination of all perturbations. Introducing systematic bias in the underlying data had no effect on RMSE.

Conclusions:

In this proof-of-concept study, performance of predictive models for cardiac competence generated from continuously-acquired physiological data was relatively stable with declining quality of the source data. As such, should these findings be replicated in other context, it follows that lower accuracy might not be an absolute contraindication for the use of consumer-oriented wearable devices in clinical prediction models.