JMIR Publications

ABSTRACT

Background:

The coronavirus disease (COVID-19) pandemic has increased the demand for telepsychiatry in neurodevelopmental assessment; however, few studies have examined the inter-rater reliability of clinician-rated autism scales, such as the Childhood Autism Rating Scale, Second Edition (CARS2), when used remotely. Although the feasibility and validity of telehealth assessments have been explored previously, the consistency between remote and in-person CARS2 ratings by different clinicians remains underinvestigated.

Objective:

To evaluate the inter-rater reliability of the CARS2 scores administered via telepsychiatry compared with in-person assessments in children with autism spectrum disorder (ASD) and/or attention-deficit/hyperactivity disorder (ADHD).

Methods:

In this randomized feasibility study, 74 children (mean age 10.4 years) with DSM-5 diagnoses of ASD, ADHD, or both were recruited from five medical institutions in Japan. Each participant underwent two CARS2 evaluations, one face-to-face and the other via telepsychiatry, by two independent evaluators who were blinded to each other’s ratings. Remote assessments were conducted using a secure video platform facilitated by the caregiver. Inter-rater reliability was assessed using intraclass correlation coefficients (ICC) stratified by diagnosis, sex, and age.

Results:

The overall ICC for CARS2 total scores between face-to-face and remote assessments was 0.582 (95% CI 0.407–0.715), indicating moderate agreement. The ICCs were slightly higher in children with ASD (0.541) than in those with ADHD (0.495). Reliability was greater among girls (ICC 0.666) than boys (0.459), and among older children (≥11 years, ICC 0.676) compared to younger children (≤10 years, ICC 0.417). Qualitative observations and prior literature suggest that these differences may reflect age- and context-related behavioral variability, challenges in observing non-verbal communication via video, and environmental distractions in home settings.

Conclusions:

Remote administration of the CARS2 via telepsychiatry yielded moderate overall inter-rater reliability with better consistency in older children and girls. Although remote tools show promise for increasing access to developmental assessments, clinicians should remain aware of context- and modality-specific limitations. Further work with larger and more diverse samples should refine the protocols and enhance standardization. Clinical Trial: The study protocol was registered with the UMIN Clinical Trials Registry (UMIN000039860).