Maintenance Notice

Due to necessary scheduled maintenance, the JMIR Publications website will be unavailable from Wednesday, July 01, 2020 at 8:00 PM to 10:00 PM EST. We apologize in advance for any inconvenience this may cause you.

Who will be affected?

Accepted for/Published in: JMIR AI

Date Submitted: Sep 29, 2025
Date Accepted: Sep 9, 2026

The final, peer-reviewed published version of this preprint can be found here:

Balancing Human and AI Features for Digital Autism Diagnostics from Home Videos: Human-In-The-Loop Machine Learning Study

Rasool A, Dunlap K, Husic A, Kline A, Azizian P, Wall D, Washington P

Balancing Human and AI Features for Digital Autism Diagnostics from Home Videos: Human-In-The-Loop Machine Learning Study

JMIR AI 2026;5:e84699

DOI: 10.2196/84699

Balancing Human and AI Intelligence for Digital Autism Diagnostics from Home Videos: Human-In-The-Loop Machine Learning Study

  • Abdur Rasool; 
  • Kaitlyn Dunlap; 
  • Arman Husic; 
  • Aaron Kline; 
  • Parnian Azizian; 
  • Dennis Wall; 
  • Peter Washington

ABSTRACT

Background:

Autism diagnosis remains hindered by specialist-dependent assessments, causing critical delays. While AI-powered video analysis promises scalability, it currently fails in many cases to capture nuanced social behaviors vital for accurate detection.

Objective:

We propose a human-in-the-loop (HITL) framework that dynamically combines deep learning with crowdsourced human intelligence to improve autism classification using real-world video data.

Methods:

We developed a HITL framework that integrates deep learning-based facial emotion (FEFE) and eye gaze (EG) feature extraction with crowdsourced human annotations for low-confidence AI predictions. The system was evaluated on 120 parent-child interaction videos from the GuessWhat? App. We studied the effect on the confidence thresholds for triggering human annotations to replace AI-based feature extraction models. Diagnostic performance for all threshold choices (0.1–0.9 in increments of 0.1) was measured via sensitivity, specificity, F1 score, and AUROC.

Results:

Including more human annotations in place of AI-generated behavior annotations led to an increase in performance. At the highest threshold (Th = 0.9; 72.2% human annotations), sensitivity increased by ∼50% and F1 score by ∼60%, while AUROC improved from <65% (AI-dominant) to >80% (human-dominant). Human annotators outperformed AI in classifying subtle emotions (e.g., Surprise, Disgust) and ambiguous cases (e.g., “No/Partial Face”), highlighting current limitations of pure automation.

Conclusions:

Human-AI integration significantly improves autism detection accuracy, particularly for nuanced social cues. This HITL approach provides a scalable solution for remote phenotyping, demonstrating that human oversight remains essential for reliable behavioral AI applications.


 Citation

Please cite as:

Rasool A, Dunlap K, Husic A, Kline A, Azizian P, Wall D, Washington P

Balancing Human and AI Features for Digital Autism Diagnostics from Home Videos: Human-In-The-Loop Machine Learning Study

JMIR AI 2026;5:e84699

DOI: 10.2196/84699

Download PDF


Request queued. Please wait while the file is being generated. It may take some time.

© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.