JMIR Publications

ABSTRACT

Background:

Chatbots are increasingly used to provide mental health support. Given their widespread accessibility and limited clinical oversight, evaluating their safety and therapeutic competence is highly relevant. However, reliable and validated methods for evaluating chatbots' ability to safely and competently respond to clinical presentations with varying severity are still lacking for non-English languages.

Objective:

1) To develop a reliable rubric grounded in evidence-based clinical practice for rating chatbot responses in terms of safety and therapeutic competence, 2) to apply it in an evaluation of chatbots' performance on Danish mental health related prompts.

Methods:

We developed 15 mental health related Danish prompts spanning five clinical domains (paranoid delusions, elevated mood, suicidal ideation, depression, and anxiety), and a rubric of 4-8 binary criteria per prompt, classified as either safety or competence criteria. Two clinicians calibrated the rubric across three training rounds before conducting a blinded, cross-sectional rating of five therapy-focused mental health chatbots and five general-purpose chatbots. Each chatbot generated two independent responses per prompt, yielding 299 responses. The two clinicians rated every response independently, blind to chatbot identity, producing 1696 criterion ratings. Disagreements were resolved by consensus, or by expert adjudication where consensus failed. Chatbot performance was primarily characterized by how often responses met 1) every safety criterion, 2) every competence criterion, 3) both, or 4) neither. Secondly, we characterized variation in performance across chatbots, chatbot type, clinical domain, presentation severity, and between each chatbot's responses to the same prompt.

Results:

Interrater reliability was substantial (κ=0.80, 95% CI 0.77-0.83). 5.0% of responses met all safety and competence criteria, and 57.5% met neither. 21.7% of responses were safe but not competent, and 15.7% were competent but not safe. The least safe responses were for paranoid delusion prompts (3.3% met all safety criteria). The least competent responses were for suicidal ideation (0% met all competence criteria). Therapy-focused chatbots directed users toward support in 6.2% of applicable responses versus 60.0% for general-purpose chatbots but also provided less harmful information. Individually, chatbots passed between 40.6% (95% CI 34.5-47.3) and 65.9% (95% CI 59.0-72.7) of all criteria. Chatbots' responses to the same prompt gave identical criterion ratings 87.5% of the time.

Conclusions:

Few responses met all predefined safety and competence criteria, and criterion fulfillment varied across chatbots and clinical domains. Performance was poorest in two high-stakes domains: suicidal ideation and paranoid delusions. Future work should establish the validity of standardized rubrics for evaluating mental health chatbots and test whether automated rating can reproduce clinician ratings for continual evaluation as future chatbots are deployed. The rubric and decision manual are available as multimedia appendices, and the rating dataset is available upon reasonable request.