JMIR Publications

ABSTRACT

Background:

Millions of people now use leading generative AI tools (chatbots) for psychological support. Despite the promise related to availability and scale, the single most pressing question in AI for mental health is whether these tools are safe. The field currently lacks a clinically validated, evidence-based benchmark for determining AI safety for mental health. The Validation of Ethical and Responsible AI in Mental Health (VERA-MH) evaluation was recently proposed to meet the urgent need for a rigorous, automated safety benchmark.

Objective:

This study aims to examine the clinical validity and reliability of the recently developed VERA-MH evaluation for AI safety in suicide risk detection and response.

Methods:

We simulated a large set of conversations between large language model (LLM)-based users (“user-agents”) spanning a wide range of suicide risk levels and disclosure styles and general-purpose AI chatbots. Licensed mental health clinicians used a rubric (scoring guide) developed for VERA-MH to independently rate the simulated conversations for safe and unsafe chatbot behaviors, as well as user-agent realism. An LLM-based evaluator (the VERA-MH “judge”) used the same scoring rubric to evaluate the same set of simulated conversations. We then compared rating alignment across (a) individual clinicians and (b) clinician consensus and the LLM judge, and (c) examined clinicians’ ratings of user-agent realism.

Results:

Individual clinicians were generally consistent with one another in their safety ratings (chance-corrected inter-rater reliability [IRR]: 0.77), thus establishing a gold-standard clinical reference. The LLM judge was strongly aligned with this clinical consensus (IRR: 0.81) overall and within key conditions. Clinician raters generally perceived the user-agents to be realistic.

Conclusions:

For the potential mental health benefits of AI chatbots to be realized, attention to safety is paramount. Findings from this human evaluation study support the clinical validity and reliability of VERA-MH: an open-source, fully automated AI safety evaluation for mental health. Further research will address VERA-MH generalizability and robustness, and future expansions can target other key areas of AI safety for mental health.