JMIR Publications

ABSTRACT

Background:

Mental health conditions are now the leading cause of disability worldwide, affecting more than one billion individuals, and suicide accounts for 720,000 to 800,000 deaths annually. While AI-native mental health chatbots have the potential to fill a sizable gap between treatment need and workforce capacity, their rapid proliferation has outpaced the development of standardized, scalable procedures to identify and respond to suicidal ideation. Despite consistent AI best practice recommendations to incorporate human oversight for safety monitoring and intervention, human-in-the-loop implementation models remain the exception rather than the rule for most generative AI mental health tools. Moreover, published studies of procedures for human clinical oversight and crisis response are scarce.

Objective:

To implement a human-in-the-loop suicide risk screening and clinical escalation protocol in a generative AI mental health chatbot.

Methods:

In a prospective implementation study of a human-in-the-loop suicide risk detection and escalation protocol, adults were recruited online to participate in a single-arm trial of an AI-native mental health chatbot. Participants completed a pre-treatment screening using the Patient Health Questionnaire-9 (PHQ-9). Individuals who endorsed the suicidal ideation item of the PHQ-9 were escalated to a human clinician for further assessment, safety planning and provision of referral resources, as appropriate. The Ask Suicide Questionnaire (ASQ) guided human-led assessment and risk stratification. Prevalence and demographic correlates of suicidal ideation were assessed, along with the rate of participant responsiveness to human outreach, time from suicidality risk detection to human response, and results of risk stratification evaluation.

Results:

All eligible participants who enrolled in the study were screened for suicide risk. Of those who completed the pre-treatment suicide screening (N=624), 81 participants (13.0%) endorsed suicidal ideation on the PHQ-9. A total of 25 of those who screened positive (31%) were reached by a human clinician by phone within 11.2 minutes (SD=13.2) for further screening and assessment, and 2 (3.6%) responded to e-mail outreach. After screening, no patients required emergency suicide interventions (100% non-acute positive). Notably, 72% (18/25) of those who screened positive reported that they were receiving concurrent mental health treatment.

Conclusions:

Proactive, universal suicide risk screening using an empirically validated measure is feasible for individuals seeking mental health support from an AI-native platform. Implementing this practice at scale has the potential to make a significant public health impact, given that the prevalence of suicidal ideation in this population is notably higher than that observed in virtual behavioral health care settings. A substantial subgroup of those who screened positive for suicidality on the PHQ-9 were responsive to human clinical outreach, supporting the feasibility and acceptability of a human-in-the-loop model of AI-supported mental health care. Clinical Trial: Exemption Status, IRB Tracking #20254520