JMIR Publications

ABSTRACT

Background:

Most patients are not at elevated risk for suicide and active interview for assessment of suicide risk may not be reasonable.

Objective:

This paper describes passive monitoring of suicide risk factors from (a) text of conversations with the AI system or (b) medical history events reported to the AI system. We report the relationship between the number of risk factors to replicate the judgment of two experts regarding elevated risk of suicide.

Methods:

The adequacy of the text monitoring system was assessed by agreement in classification of risk factors by the LLM and a trained human reviewer, in 18 dialogues and 48 case descriptions; found in two books typically used to train clinicians on detection of suicide risks. The classification of medical history events into suicide risk factors was done using likelihood of suicide reported in the literature. Two clinicians rated relative risk levels of 86 pairs of dialogues/case descriptions. An index of risk was derived from these pairwise comparisons. This paper reports the percentage of variation in expert-assessed risk of suicide explained by the number of risk factors identified by the LLM.

Results:

On average the human rater and the LLM agreed in 97.03% (standard deviation of 3.98%) of risk factors in the 66 dialogues/case descriptions. The count of risk factors explained 30% of the variation in experts’ rating of the risk of suicide and achieved near-perfect accuracy (AROC =99%) in classifying dialogues into low vs high/moderate risk groups.

Conclusions:

The proposed method of counting risk factors for suicide could be a reasonable strategy to passively monitor risk of suicide in AI conversations with depressed patients. Additional research is needed to compare the performance of this passive method compared to active suicide assessment.