JMIR Publications

ABSTRACT

Background:

Mental health misconceptions are widespread, yet little is known about how accurately psychology students identify them, how accuracy varies across stages of psychology education, and how students’ performance compares with that of large language models (LLMs)

Objective:

To assess how accurately psychology students at different stages of their education and four LLMs distinguish mental health myths from facts and to examine response patterns across individual statements.

Methods:

We evaluated the accuracy of 150 psychology students, including 70 first-semester and 80 more advanced students, in classifying 15 mental health myths and 15 facts. Additionally, we used ChatGPT-5.4, Claude Sonnet 4.5, DeepSeek V3.2, and GLM-5 to classify the same statements across 80 independent runs per statement and model.

Results:

First-semester students classified 75.2% of myths and 55.4% of facts accurately; corresponding values were 87.7% and 69.3% for advanced students. A binomial generalized linear mixed-effects model indicated that group differences in classification accuracy varied between myths and facts. Compared with first-semester students, advanced students had higher odds of accurate classification for both myths (odds ratio [OR] 3.22, 95% CI 2.40-4.32; P<.001) and facts (OR 2.83, 95% CI 2.17-3.69; P<.001). LLMs generally showed higher accuracy than both student groups in classifying statements, although performance varied substantially across models, particularly for facts. Item-level analyses revealed partly distinct error patterns: students and LLMs struggled with some of the same facts, whereas they also showed relative advantages on specific statements.

Conclusions:

These findings suggest that mental health literacy differs significantly across stages of psychology education and that, although current LLMs perform strongly on this classification task, some item-specific errors recurred across repeated runs. Examining which myths and facts remain difficult may help identify specific gaps in both psychology education and AI-generated mental health information.