Accepted for/Published in: JMIR Formative Research
Date Submitted: Feb 2, 2024
Date Accepted: Feb 21, 2025
Warning: This is an author submission that is not peer-reviewed or edited. Preprints - unless they show as "accepted" - should not be relied on to guide clinical practice or health-related behavior and should not be reported in news media as established information.
Does Exposure to ChatGPT Change How Users Interact with a Motivational-Interviewing Chatbot? An Experimental Study
ABSTRACT
Background:
The worldwide introduction of ChatGPT in November 2022 may have changed how its users perceive and interact with other chatbots. This possibility may confound the comparison of responses to pre-ChatGPT and post-ChatGPT iterations of pre-existing chatbots, in turn affecting the direction of their evolution. Before the release of ChatGPT, we created a therapeutic chatbot, MIBot, whose goal is to use Motivational Interviewing (MI) to guide smokers towards making the decision to quit smoking. We were concerned that measurements going forward would not be comparable to those in the past, impacting the evaluation of future changes to the chatbot.
Objective:
Explore changes in how users interact with MIBot after the release of ChatGPT, and quantify the impact as a function of users’ familiarity with ChatGPT.
Methods:
We compared user interactions with MIBot prior to ChatGPT’s release and 6 months after the release. Participants (n=143) were recruited through an online platform in November of 2022, prior to the release of ChatGPT, to converse with MIBot, in an experiment we refer to as MI v5.2. In May of 2023, a set of (n=129) different participants were recruited to interact with the same version of MIBot, and asked additional questions about their familiarity with ChatGPT, in the experiment called MI v5.2A. We fit a logistic regression model for “ChatGPT Exposure”. We also fit two linear regression models for “Familiarity with ChatGPT”, one with MI v5.2A only and another with both cohorts, on relevant effect variables.
Results:
Fully 64% of the MI v5.2A cohort had used ChatGPT, with 51% using it on a regular basis. In the logistic regression model, ChatGPT exposure is negatively correlated to satisfaction with MIBot (β = -1.8, P = .005). In both linear regression models, familiarity with ChatGPT is positively correlated to change in confidence to quit smoking (MI v5.2A only: β = 4.1, P < .001; both cohorts: β = 1.7, P = .002) and average response length (MI v5.2A only: β = 1.0, P = .02; both cohorts: β = 0.73, P = .010), and negatively correlated with satisfaction with MIBot (MI v5.2A only: β = -16, P = .03; both cohorts: β = -14, P < .001).
Conclusions:
The widespread reach of ChatGPT has changed how users interact with MIBot. Users exposed to ChatGPT are less satisfied with MIBot in general. Moreover, users with a higher familiarity with ChatGPT offer longer responses (especially to Yes/No questions), are more strongly influenced by MIBot, and are even less satisfied with MIBot. Clinical Trial: N/A
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.