Accepted for/Published in: JMIR AI
Date Submitted: Dec 13, 2024
Open Peer Review Period: Mar 3, 2025 - Apr 28, 2025
Date Accepted: Feb 24, 2026
(closed for review but you can still tweet)
Warning: This is an author submission that is not peer-reviewed or edited. Preprints - unless they show as "accepted" - should not be relied on to guide clinical practice or health-related behavior and should not be reported in news media as established information.
Large Language Models (LLM)-Powered Diagnostic Co-pilot (“CapyEngine”) for Mental Disorders: Development, Evaluation, and Future Optimization
ABSTRACT
Background:
Despite the growing potential of large language models (LLMs) in mental health services, their application in diagnostic processes remains limited.
Objective:
This study described the development and evaluation of CapyEngine, an LLM-powered diagnostic tool designed to assist in the diagnosis of mental disorders.
Methods:
We developed and evaluated CapyEngine through three phases. In Phase 1, we created a symptom database using Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition, Text Revision (DSM-5-TR). We then developed CapyEngine's architecture using LLMs, embedding models, and vector searches. In Phase 2, we conducted interviews and usability tests with mental health professionals (n = 7) to identify challenges in traditional diagnostic practices and potential areas for CapyEngine's application. In Phase 3, we compared CapyEngine's diagnostic accuracy against ChatGPT-4 and clinicians using 35 standardized case scenarios test questions from psychiatry and clinical psychology board exams. Questions were input into CapyEngine and top 10 recommended diagnoses were obtained. ChatGPT-4 was prompted to provide the top ten potential diagnoses for each questions. Clinicians (n = 3) received similar instruction to generate at least 10 potential diagnoses for each question. Responses were then analyzed to determine accuracy within the top 10, top 5, and top 1 diagnoses.
Results:
CapyEngine achieved 62.86% accuracy for identifying correct diagnoses within the top 10 options, and 48.57% accuracy for top diagnosis. ChatGPT-4 showed 100% accuracy within top 10 and top 5 options, but only 31.43% for top diagnosis. Clinicians outperformed both AI models with 82.86% accuracy within top 10 and 57.14% for top diagnosis.
Conclusions:
CapyEngine shows promise in augmenting the mental health diagnostic process. Future enhancements will focus on incorporating non-symptom-based diagnostic factors, developing specialized embedding models, and addressing cultural sensitivity. Further research is needed to assess the risks and benefits of integrating AI tools like CapyEngine into clinical workflows and to address barriers to adoption.
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.