Balancing Human and AI Intelligence for Digital Autism Diagnostics from Home Videos: Human-In-The-Loop Machine Learning Study
ABSTRACT
Background:
Autism diagnosis remains hindered by specialist-dependent assessments, causing critical delays. While AI-powered video analysis promises scalability, it currently fails in many cases to capture nuanced social behaviors vital for accurate detection.
Objective:
We propose a human-in-the-loop (HITL) framework that dynamically combines deep learning with crowdsourced human intelligence to improve autism classification using real-world video data.
Methods:
We developed a HITL framework that integrates deep learning-based facial emotion (FEFE) and eye gaze (EG) feature extraction with crowdsourced human annotations for low-confidence AI predictions. The system was evaluated on 120 parent-child interaction videos from the GuessWhat? App. We studied the effect on the confidence thresholds for triggering human annotations to replace AI-based feature extraction models. Diagnostic performance for all threshold choices (0.1–0.9 in increments of 0.1) was measured via sensitivity, specificity, F1 score, and AUROC.
Results:
Including more human annotations in place of AI-generated behavior annotations led to an increase in performance. At the highest threshold (Th = 0.9; 72.2% human annotations), sensitivity increased by ∼50% and F1 score by ∼60%, while AUROC improved from <65% (AI-dominant) to >80% (human-dominant). Human annotators outperformed AI in classifying subtle emotions (e.g., Surprise, Disgust) and ambiguous cases (e.g., “No/Partial Face”), highlighting current limitations of pure automation.
Conclusions:
Human-AI integration significantly improves autism detection accuracy, particularly for nuanced social cues. This HITL approach provides a scalable solution for remote phenotyping, demonstrating that human oversight remains essential for reliable behavioral AI applications.
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.