JMIR Publications

ABSTRACT

Background:

A formal autism diagnosis can be an inefficient and lengthy process. Families may wait months or longer before receiving a diagnosis for their child despite evidence that earlier intervention leads to better treatment outcomes. Digital technologies which detect the presence of behaviors related to autism can scale access to pediatric diagnoses. A strong indicator of the presence of autism is self-stimulatory behaviors like hand flapping, head banging, and spinning.

Objective:

This work aims to demonstrate the feasibility of deep learning technologies for detecting hand flapping from unstructured home videos as a first step towards validating whether models and digital technologies can be leveraged to aid with autism diagnoses. However, using computer vision to detect hand flapping is especially difficult due to the lack of training data in this space and excessive camera shakiness and motion inherent in the collection of video data.

Methods:

We used the Self-Stimulatory Behavior Dataset (SSBD), which contains 75 videos of hand flapping, head banging, and spinning exhibited by children. From all the hand flapping videos, we extracted 100 positive and 100 control videos of hand flapping, each between 2 to 5 seconds in duration. We designed a custom feature representation to keep the model lightweight to enable deployment into clinical mobile therapeutics. We extracted the coordinates of hand landmarks and fed them into a Long Short-Term Memory (LSTM) network predicting the presence of hand flapping in each video clip.

Results:

Our best model achieves an F1 score of 75.2 ± 0.6 using 5-fold cross-validation on SSBD data. This model contains only 48,961 parameters, in contrast to the mobile-optimized MobileNetV2 convolutional neural network architecture with over 3.5 million model parameters.

Conclusions:

We created a novel feature representation for a lightweight neural network which can detect hand flapping. This work demonstrates that through careful feature engineering, efficient models for autism diagnosis can become feasible for use in digital therapeutic pipelines.