Previously submitted to: Journal of Medical Internet Research (no longer under consideration since Oct 07, 2021)
Date Submitted: Apr 17, 2021
Warning: This is an author submission that is not peer-reviewed or edited. Preprints - unless they show as "accepted" - should not be relied on to guide clinical practice or health-related behavior and should not be reported in news media as established information.
Agreement and Reliability Analysis of Machine Learning Scaling in the Assessment of Acute Proximal Weakness by Experts and Non-Experts: Clinical Feasibility Study
ABSTRACT
Background:
Assessing the symptoms of proximal weakness caused by neurological deficits requires expert knowledge and experienced neurologists. Recent advances in artificial intelligence and the Internet of Things have resulted in the development of automated systems that emulate physicians’ assessments.
Objective:
This study provides an agreement and reliability analysis of using an automated scoring system to evaluate proximal weakness by experts and non-experts.
Methods:
We collected 144 observations from acute stroke patients in a neurological intensive care unit to measure the symptom of proximal weakness of upper and lower limbs. A neurologist performed a gold standard assessment and two medical students performed identical tests as non-expert assessments for manual and machine learning-based scaling of Medical Research Council (MRC) proximal scores. The system collects signals from sensors attached on patients’ limbs and trains a machine learning assessment model using the hybrid approach of data-level and algorithm-level methods for the ordinal and imbalanced classification in multiple classes. For the agreement analysis, we investigated the percent agreement of MRC proximal scores and Bland-Altman plots of kinematic features between the expert- and non-expert scaling. In the reliability analysis, we analysed the intra-class correlation coefficients (ICCs) of kinematic features and Krippendorff’s alpha of the three observers’ scaling.
Results:
The mean percent agreement between the gold standard and the non-expert scaling was 0.542 for manual scaling and 0.708 for IoT-assisted machine learning scaling, with 30.63% enhancement. The ICCs of kinematic features measured using sensors ranged from 0.742 to 0.850, whereas the Krippendorff’s alpha of manual scaling for the three observers was 0.275. The Krippendorff’s alpha of machine learning scaling increased to 0.445, with 61.82% improvement.
Conclusions:
Automated scaling using sensors and machine learning provided higher inter-rater agreement and reliability in assessing acute proximal weakness. The enhanced assessment supported by the proposed system can be utilized as a reliable assessment tool for non-experts in various emergent environments.
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.