JMIR Publications

ABSTRACT

Background:

: Natural language processing and large language model systems are increasingly used to support mental health documentation, screening, and follow-up planning. In counseling contexts, model outputs may influence diagnostic framing, risk recognition, and clinical record content. Static performance metrics and fluent generated summaries are not sufficient to support safe implementation without governance, safety gating, human review, and monitoring.

Objective:

This study aimed to develop a human-governed clinical informatics framework for safe AI-assisted mental health counseling and to make the formative evidence base and requirements-mapping process traceable.

Methods:

We conducted a secondary framework-development and requirements-mapping study using the Korean AI Hub Psychological Counseling Dataset, official data-description and utilization documents, released KLUE-BERT risk-prediction model materials, released KoAlpaca summary-generation resources, and a de-identified 139-case rule-based summary safety-screening audit table derived from the original summary-comparison file. Raw counseling transcript text, reference-summary full text, and generated-summary full text were not included in the manuscript or supplementary materials. We extracted failure modes from documented data/model characteristics, released code/configuration files, documentation-reported model metrics, and rule-based proxy flags. Each failure mode was mapped to safety controls, operational criteria, and deployment-level requirements.

Results:

The official documents described 1,661 counseling sessions and 465,474 paragraph-level tokens across depression, anxiety disorder, addiction, and normal-control groups. The documented split included 1,339 training, 173 validation, and 149 test sessions. The summary-generation materials documented 1,278 training summaries and 139 test summaries. Documentation-reported model metrics included KLUE-BERT accuracies of 71.43% for depression, 73.53% for anxiety, and 66.67% for addiction, and KoAlpaca BERTScore precision, recall, and F1 values of 62.13%, 59.56%, and 60.80%, respectively. The 139-case screening table contained 77 depression, 31 anxiety, and 31 addiction cases. Rule-trigger rates included unsupported-content proxy flags in 57/139 cases (41.0%), overdiagnostic-expression proxy flags in 44/139 cases (31.7%), medicalized-expression proxy flags in 76/139 cases (54.7%), and any rule-based proxy flag in 127/139 cases (91.4%). These values are conservative rule-trigger rates rather than confirmed clinical error rates. The findings informed a 7-stage workflow, 6 safety-control layers, an operational safety gate, a workflow-to-control crosswalk, deployment-level transition criteria, and a constructed high-risk example.

Conclusions:

AI-assisted mental health counseling should be implemented as a governed clinical information workflow rather than as an autonomous diagnostic or documentation pathway. The proposed framework specifies safeguards and validation requirements for future supervised evaluations, but it does not itself establish clinical safety or clinical effectiveness. Prospective simulation, clinician usability testing, patient/client feedback, and independent expert validation remain necessary before routine deployment.