JMIR Publications

ABSTRACT

Background:

Early identification of Major Depressive Disorder (MDD) and the full spectrum of self-harm and suicidality—specifically suicidal ideation (SI), attempts (SA), and non-suicidal self-injury (SH)—is often impeded by the resource intensity of psychiatric interviews. While AI-driven conversational screeners offer scalable triage, their clinical utility hinges on rigorous validation against criterion diagnostic assessments in real-world settings

Objective:

This study aims to empirically evaluate the diagnostic concordance between a multimodal AI mental health screener and the Mini-International Neuropsychiatric Interview (MINI), serving as the criterion standard, for Major Depressive Disorder (MDD) and the full spectrum of self-harm and suicidality (including suicidal ideation [SI], suicide attempts [SA], and non-suicidal self-injury [SH]) within an outpatient clinical cohort. The research will comprehensively validate the AI tool's clinical efficacy and feasibility by quantifying Cohen's Kappa for diagnostic agreement, detailing the error profiles of multi-category risk assessments, and analyzing the influence of interview duration on agreement and subgroup robustness.

Methods:

We conducted a prospective, cross-sectional diagnostic agreement study in the outpatient department of a tertiary psychiatric hospital. A convenience sample of 75 adolescents and young adults presenting for routine assessment was enrolled. Each participant first completed an automated conversational screening interview delivered through a voice-interactive device, followed immediately by a clinician-administered Mini-International Neuropsychiatric Interview for Children and Adolescents (MINI-KID); interviewers were blinded to the screening results. The screener generated binary classifications for major depressive disorder (MDD) and suicide-related outcomes (suicidal ideation, suicide attempts, and self-harm), which were prespecified to correspond to MINI-KID diagnoses coded according to DSM-IV/ICD-10 criteria.Primary validation outcomes were percent positive agreement (PPA), percent negative agreement (NPA), overall agreement (POA), and Cohen’s κ with 95% confidence intervals; McNemar tests were used to assess directional disagreement. Secondary analyses examined agreement stratified by sex and age group and explored whether interview duration was associated with discrepant classifications. In addition, we developed a descriptive error taxonomy to characterize patterns of false-positive and false-negative classifications.

Results:

The final agreement analysis included 74 participants (with an effective n per endpoint ranging from 74 to 75 due to one system-level missing case); the cohort was predominantly female (71.6%) with a mean age of 32.5 years, consistent with a young adult tertiary outpatient psychiatric population. The MINI reference standard indicated a high burden of target conditions, with major depressive disorder (MDD) diagnosed in 67.6% of participants, clinically significant suicidal ideation (SI) in 64.9%, a history of suicide attempts or plans (SA) in 36.5%, and non-lethal self-harm (SH) in 56.8%. Across all endpoints, the conversational screener showed substantial to almost perfect agreement with MINI diagnoses: for SI, percent positive agreement (PPA) was 93.8%, percent negative agreement (NPA) was 85.2%, and overall agreement (POA) was 90.7%, corresponding to a Cohen’s κ of 0.80 (95% CI 0.65–0.94) with no evidence of directional imbalance in misclassification (McNemar P=1.00). Agreement for higher-acuity suicidal outcomes was similarly strong: for SA, the screener achieved a PPA of 100%, an NPA of 93.3%, and a POA of 96.0%, yielding a κ of 0.92 (95% CI 0.83–1.01), while for SH, PPA was 92.5%, NPA was 100%, and POA was again 96.0% with κ=0.92 (95% CI 0.83–1.01). For MDD, PPA was 94.0% and NPA was 83.3%, resulting in a POA of 90.5% and a κ of 0.78 (95% CI 0.63–0.94), and discordant pairs were balanced (McNemar P=1.00). A structured error audit indicated that discrepancies were confined to a small number of false positives—often in individuals with prominent impulsivity or atypical language patterns—and false negatives, typically in interviews characterized by brief or minimal disclosure of ideation, without statistically significant directional bias across endpoints. Subgroup analyses stratified by age, sex, and interview setting did not reveal material fairness gaps in diagnostic performance, and interview duration was not significantly associated with disagreement, while most participants were classified within moderate risk bands, aligning with expectations for a tertiary psychiatric outpatient sample.

Conclusions:

In conclusion, this prospective clinical validation study demonstrates that a AI conversational screener exhibits high diagnostic concordance with the clinician-administered Mini-International Neuropsychiatric Interview (MINI) for assessing Major Depressive Disorder (MDD) and suicide-related outcomes in a routine outpatient setting. The quantified Cohen's Kappa (K) coefficients ranged from 0.78 to 0.92, consistently exceeding the established threshold for substantial agreement (K > 0.61), with the agreement for high-acuity risks (SA/SH) reaching an exceptionally high level (K=0.92). These robust findings strongly suggest that the AI assessment can serve as a highly effective, safe supplement to clinician-administered structured interviews, thereby reducing assessment burden on specialists while maintaining the requisite standard for diagnostic accuracy and patient safety. To definitively confirm its utility, larger, multi-site studies are necessary, focusing on external validation across diverse populations and ensuring cultural and linguistic adaptation to confirm the generalizability of these promising results.