JMIR Publications

ABSTRACT

Background:

Large language models (LLMs) such as OpenAI’s GPT-3.5, GPT-4, and GPT-4o have garnered early and significant enthusiasm for their potential applications within mental health, ranging from documentation support to chat-bot therapy. Understanding the accuracy and reliability of the psychiatric parametric knowledge encoded in these models, and developing measures of confidence in their responses (i.e., the likelihood that an LLM response is accurate) is crucial for safe and effective integration of these tools into mental health settings.

Objective:

To assess the accuracy, reliability, and predictors of accuracy of GPT-3.5, GPT-4, and GPT-4o on standardized psychiatry multiple-choice questions (MCQs).

Methods:

A cross-sectional study was conducted where three commonly available, commercial LLMs (GPT-3.5, GPT-4, and GPT-4o) were tested for their ability to provide answers to 150 single-answer MCQs extracted from the Psychiatry Test Preparation and Review Manual. Each model generated answers to every MCQ ten times. We evaluated the accuracy and reliability of the answers and sought predictors of answer accuracy. Our primary outcome was the proportion of questions answered correctly by each LLM (accuracy). Secondary measures were (i) response consistency to MCQs across ten trials (reliability), (ii) the correlation between MCQ answer accuracy and response consistency, and (iii) the correlation between MCQ answer accuracy and the models’ self-reported confidence.

Results:

GPT-3.5 answered 58% of MCQs correctly on first attempt, while GPT-4 and GPT-4o achieved 84% and 87% accuracy, respectively; GPT-4 and GPT-4o displayed no difference in performance, but significantly outperformed GPT-3.5 (p < .010). GPT-3.5 exhibited less response consistency on average compared to the other models (p < .010). MCQ response consistency was positively correlated with MCQ accuracy across all models (p < .001), whereas model self-reported confidence showed no correlation with accuracy except for GPT3.5, where self-reported confidence was weakly inversely correlated with correctness (p < .010).

Conclusions:

To our knowledge, this is the first comprehensive evaluation of the psychiatric knowledge encoded in commercially available LLMs and the first to assess their reliability and identify predictors of response accuracy within medical domains. Findings suggest GPT-4 and GPT-4o encode accurate and reliable psychiatric knowledge, and that methods such as repeated prompting may provide a measure of LLM response confidence. This work supports the potential of LLMs in mental health settings and motivates further research to assess their performance in more open-ended clinical contexts.