Currently submitted to: JMIR Medical Informatics
Date Submitted: Nov 16, 2025
Open Peer Review Period: Nov 26, 2025 - Jan 21, 2026
(closed for review but you can still tweet)
NOTE: This is an unreviewed Preprint
Warning: This is a unreviewed preprint (What is a preprint?). Readers are warned that the document has not been peer-reviewed by expert/patient reviewers or an academic editor, may contain misleading claims, and is likely to undergo changes before final publication, if accepted, or may have been rejected/withdrawn (a note "no longer under consideration" will appear above).
Peer review me: Readers with interest and expertise are encouraged to sign up as peer-reviewer, if the paper is within an open peer-review period (in this case, a "Peer Review Me" button to sign up as reviewer is displayed above). All preprints currently open for review are listed here. Outside of the formal open peer-review period we encourage you to tweet about the preprint.
Citation: Please cite this preprint only for review purposes or for grant applications and CVs (if you are the author).
Final version: If our system detects a final peer-reviewed "version of record" (VoR) published in any journal, a link to that VoR will appear below. Readers are then encourage to cite the VoR instead of this preprint.
Settings: If you are the author, you can login and change the preprint display settings, but the preprint URL/DOI is supposed to be stable and citable, so it should not be removed once posted.
Submit: To post your own preprint, simply submit to any JMIR journal, and choose the appropriate settings to expose your submitted version as preprint.
Warning: This is an author submission that is not peer-reviewed or edited. Preprints - unless they show as "accepted" - should not be relied on to guide clinical practice or health-related behavior and should not be reported in news media as established information.
From Evaluation to Enhancement: A Decision Support Framework for Quality Assurance in Therapeutic AI Systems
ABSTRACT
Background:
Therapeutic chatbots are increasingly deployed across digital mental health services, yet most evaluation efforts remain diagnostic rather than actionable. As a result, organizations lack structured pathways to translate evaluation findings into validated quality improvements suitable for clinical governance.
Objective:
This study introduces EvaluationPlus, a decision support framework that operationalizes a reproducible evaluation→enhancement loop for therapeutic AI systems. We aimed to demonstrate its feasibility through expert-guided diagnosis, multi-LLM enhancement mapping, and within-subject validation.
Methods:
Using the bilingual mental health chatbot Dr.CareSam, we conducted three iterative enhancement cycles. Two clinical psychologists performed structured diagnostic reviews using think-aloud protocols to identify competency-specific deficits. Three large language models (GPT-4.0, Claude 4.0 Sonnet, Gemini 2.5) generated prescriptive enhancement strategies aligned with a seven-dimension therapeutic competency rubric. A double-blind, within-subject A/B study with Korean university students (N=15; IRB-approved) compared baseline and enhanced versions across standardized scenarios.
Results:
Enhancement cycles yielded substantial overall improvement in therapeutic quality (+21%; 6.26→7.59; dz=0.71). Targeted dimensions (Active Listening and Questioning, Personalization, Complex Thinking) improved by +3.04 points, exceeding gains observed in non-targeted domains (+1.62). User preference strongly favored the enhanced system (87%), and expert ratings confirmed maintained safety and therapeutic appropriateness.
Conclusions:
EvaluationPlus provides a governance-ready framework for continuous quality assurance of therapeutic AI systems. By linking expert diagnostic procedures with prescriptive LLM-driven enhancements and multi-stakeholder validation, the framework supports reproducible improvement cycles suitable for clinical deployment. Clinical Trial: Not applicable. This study was a non-clinical performance-evaluation experiment, not a clinical trial.
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.