Currently submitted to: JMIR AI
Date Submitted: Nov 6, 2026
Open Peer Review Period: Nov 17, 2026 - Jan 12, 2027
(currently open for review)
Warning: This is an author submission that is not peer-reviewed or edited. Preprints - unless they show as "accepted" - should not be relied on to guide clinical practice or health-related behavior and should not be reported in news media as established information.
Multimodal Foundation Models That Describe Autism-Related Behaviors in Children's Videos for Caregivers: Stage-Wise Benchmark Study
ABSTRACT
Background:
AI-generated reports could help caregivers understand their children's behavior, but a clearly written report may still be based on a mistaken observation. For example, a child who remains silent may be described as nonresponsive to speech even when no one has addressed them. Errors can arise both when a model interprets the video and when it turns those observations into a report. Evaluating the final report alone makes it difficult to determine where these errors began.
Objective:
We compared how well AI models label autism-related behaviors in children's videos and turn observations into caregiver reports. We then examined whether revising the prompts and separating video observation from report writing could improve performance.
Methods:
We screened the retrievable clips from the audio-visual autism spectrum dataset (AV-ASD) for technical quality, retaining 701 clips annotated for nine autism-related behaviors and a Background category indicating that none of these behaviors was labeled. We compared five video-capable models against the reference labels using the 578 clips for which all five returned complete labels. Five language models wrote caregiver reports from the same observation records. The report comparison included 455 clips with complete rubric-item scores from all three AI judges for every model. To understand the models' errors, one study author trained in child development and autism research reviewed 50 clips alongside the model outputs and reference labels. We then compared original and revised prompts for behavior labeling on 134 evaluation clips and report writing on 95 development clips. Finally, we examined whether generating an observation record before writing a report improved results compared with writing directly from video, under matched instructions.
Results:
Gemini 3.7 Flash achieved the highest macro-F1 across the 10 labels (0.671, 95% CI 0.637 to 0.701). The review identified eight error types, four of which concerned whether a response had been invited or could be observed. With revised prompts, nine-label macro-F1 increased from 0.607 to 0.707 (difference 0.100, 95% CI 0.058 to 0.139), and the judge-rated report score from 0.880 to 0.987 (difference 0.106, 95% CI 0.093 to 0.120). Separating observation from report writing also produced higher ratings on every judge-rated dimension (composite 0.961 vs 0.930; difference 0.031, 95% CI 0.023 to 0.037). However, these two-stage reports showed no improvement in nine-label macro-F1 and mentioned more behaviors that the annotators had not labeled (0.44 vs 0.19 per report; difference 0.25, 95% CI 0.14 to 0.35).
Conclusions:
Prompt revisions improved agreement with reference labels and AI-rated report quality. Two-stage reports also received higher ratings from judges who did not see the video, although their agreement with the reference labels did not improve. For behaviors defined by a missing response, evaluation must establish whether a response was invited and could be observed.
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.