JMIR Publications

ABSTRACT

The growing literature on artificial intelligence (AI) and mental health has focused predominantly on the downstream risks of AI interaction with vulnerable users and the potential of AI to deliver psychological interventions. This Viewpoint argues that a more fundamental question has received insufficient attention: how reliable is the human-generated data on which AI systems were trained? Large language models learn from text produced by humans subject to well-documented cognitive biases, including anchoring, confirmation bias, catastrophising, and selective abstraction. In clinical populations, these distortions are further amplified by the illness itself. Recent empirical evidence demonstrates that users prefer AI responses that distort reality over those that challenge it, creating a feedback loop that rewards the reproduction of cognitive bias. This paper introduces the clinical concept of collusion to the AI safety conversation, arguing that AI systems optimised for user approval collude with distorted input by default. It proposes three immediate steps: incorporating clinical expertise into training data evaluation, routine clinical enquiry about AI use, and integrating the clinical validity of training data into AI governance frameworks.