JMIR Publications

ABSTRACT

Background:

Trauma exposure is highly prevalent and associated with various health issues. However, healthcare professionals can exhibit trauma-related diagnostic overshadowing bias, leading to misdiagnosis and inadequate treatment of trauma-exposed populations. Generative artificial intelligence (GAI) models are increasingly used in healthcare contexts. No research has examined whether generative artificial intelligence (GAI) demonstrates this bias in decision-making, and how rates of this bias may compare to mental health professionals (MHPs).

Objective:

This study aimed to 1) assess trauma-related diagnostic overshadowing among frontier GAI models, and 2) compare evidence of trauma-related diagnostic overshadowing between frontier GAI models and MHPs.

Methods:

Mental health professionals (N = 232, M age = 43.7) completed an experimental paradigm consisting of two vignettes describing adults presenting with obsessive-compulsive symptoms or substance abuse symptoms. One vignette included a trauma exposure history (i.e., sexual trauma or physical trauma), and one vignette did not include a trauma exposure history. Participants answered questions about their preferences for diagnosis and treatment options for clients within the vignettes. GAI models (Gemini 1.5 Flash, ChatGPT 4o mini, Meta Llama 3) completed the same experimental paradigm, with each block being reviewed by each GAI model 20 times. Independent samples t-tests and chi-square analyses were used to assess diagnostic and treatment decision-making across vignette factors and respondents.

Results:

Similarly to MHPs, GAI models demonstrated some evidence of trauma-related diagnostic overshadowing bias, particularly in Likert ratings for PTSD diagnosis and treatment when sexual trauma was present (p < .001). However, GAI models mostly exhibited less bias than MHPs in both Likert and forced-choice clinical decisions. Compared to MHPs, GAI models assigned significantly greater mean target diagnosis and treatment ratings for OCD cases (p < .001) and target treatment ratings for SUD cases (p < .001) when trauma was present. In forced-choice selections, GAI models were significantly more accurate than MHPs in OCD cases and SUD cases with sexual trauma (p < .001).

Conclusions:

GAI models demonstrate evidence of trauma-related diagnostic overshadowing bias, yet the degree of bias varies by task and model. However, GAI models generally demonstrated less bias than mental health professionals in this experimental paradigm. These findings highlight the importance of understanding GAI biases in mental healthcare. More research into bias reduction strategies and responsible implementation of GAI models in mental healthcare is needed.