JMIR Publications

ABSTRACT

Background:

Falls are frequent among long-term care residents. Machine-learning (ML) based fall prediction algorithms may facilitate early intervention; however, we need to understand whether they benefit or hinder clinical decision-making before clinical adoption.

Objective:

This study aimed to 1) determine how a ML fall prediction algorithm influences physiotherapists’ prediction of future falls and 2) explore how the ML algorithm influences physiotherapists’ confidence in future fall prediction.

Methods:

Forty-four registered physiotherapists with experience in geriatrics participated in a virtual observational study. Physiotherapists provided a prediction of the percent likelihood of a long-term care resident (n=5; 3 ratings per resident) falling in the next month, before and after reviewing the output of a fall prediction algorithm and rated their confidence in their prediction. All predictions were compared to the residents’ actual future falls to confirm discriminative ability pre- and post-algorithm. An ROC curve was used to show the performance of the physiotherapist (pre-algorithm) and physiotherapist (post-algorithm) in their ability to discriminate between fallers and non-fallers. Linear mixed models were used to examine factors associated with accuracy pre- and post-algorithm and linear regressions were used to understand changes in physiotherapist confidence pre- and post-algorithm. Finally, content analysis was used to analyze physiotherapist feedback.

Results:

There was an increase in physiotherapists’ ability to discriminate between a resident falling and not falling within the next four weeks after reviewing the results of the algorithm (pre-algorithm area under curve (AUC):0.78; 95% confidence interval (CI):0.49-1; post-algorithm AUC:0.86; 95% CI:0.61-1). Physiotherapist fall predictions moved closer to the algorithm after using it in their decision-making (b=-7.2; p<0.001), even in instances where the algorithm was wrong. There was no effect of age, confidence, and years of clinical experience on physiotherapist (pre-algorithm) accuracy, however, there was an effect of confidence and age on physiotherapist (post-algorithm) accuracy. Most (33/44) physiotherapists reported that they would use a fall prediction algorithm to complement their assessments, but many had concerns about algorithmic accuracy and transparency.

Conclusions:

ML-based prediction models may improve a physiotherapist’s fall risk assessment but they do not improve their decisional confidence and there is a risk of automation bias. Future research is needed into how to best incorporate information from these algorithms into clinical decision-making around fall risk and address issues of safety and accountability.