JMIR Publications

ABSTRACT

Background:

As generative AI (GenAI) is increasingly used in behavior interventions, ensuring the safety and appropriateness of AI-generated statements and detecting high-risk patient disclosures are critical.

Objective:

We developed and tested SafetyScan, a large language model (LLM)-assisted transcript scanner designed to support human-on-the-loop (HOTL) oversight by identifying and categorizing potentially concerning content in counseling encounters.

Methods:

To validate SafetyScan, we analyzed transcripts from a completed study where a GenAI conducted motivational interviewing with young adults reporting hazardous alcohol consumption. We analyzed 37 transcripts previously confirmed by human raters to contain no concerning content (serving as negative controls). We then created 37 positive test cases by inserting one randomly generated concerning statement into each of these original transcripts, drawn from five predefined categories: client safety (e.g., suicidal ideation), unsupported treatment recommendation, inappropriate counselor comment, discriminatory language, and invented client history. SafetyScan analyzed all 74 transcripts.

Results:

SafetyScan demonstrated strong performance with a sensitivity of 0.92, specificity of 1.00, and an F1 score of 0.96. It correctly identified 34 of the 37 inserted concerning statements (true positives) and correctly cleared all 37 benign transcripts (true negatives), resulting in 3 false negatives and 0 false positives. The three missed statements were two inappropriate counselor responses and one unsupported treatment recommendation, none judged as critical safety failures.

Conclusions:

These findings suggest SafetyScan offers a feasible and accurate solution for assisting human oversight of GenAI-driven counseling, supporting safer deployment of conversational AI systems in clinical and research contexts. Clinical Trial: NA