JMIR Publications

ABSTRACT

Background:

Artificial intelligence (AI) chatbots have become prominent tools in healthcare to enhance health knowledge and promote healthy behaviors across diverse populations. However, factors influencing the perception of AI chatbots and human-AI interaction are largely unknown.

Objective:

To identify interaction characteristics associated with the perception of an AI chatbot identity as a human versus an artificial agent, adjusting for sociodemographic status and prior chatbot use in a diverse sample of women.

Methods:

This was a secondary analysis of data from the HeartBot study in women aged 25 years or older. The goal of the HeartBot was to evaluate the change in awareness of heart disease after interacting with a fully automated AI chatbot. Women were recruited through social media from October 2023 to January 2024. The perceived chatbot identity (human vs. artificial agent), conversation length with the HeartBot, message humanness, message effectiveness, and attitude toward AI were measured at the post-chatbot survey. Multivariate logistic regression was conducted to explore factors predicting women’s perception of a chatbot's identity as a human, adjusting for age, race/ethnicity, education, prior AI chatbot use, message humanness, message effectiveness, and attitude towards AI.

Results:

Among 92 women (mean age 45.9 years, SD 11.9; range 26–70), the chatbot identity was correctly identified by two-thirds (66%) of the sample, while one-third (34%) misidentified the chatbot as a human. Over half (58%) had prior AI chatbot experience. On average, participants interacted with the HeartBot for 13.0 minutes (SD 7.8) and entered 82.5 words (SD 61.9). In multivariable analysis, only message humanness was significantly associated with the perception of chatbot identity as a human compared to an artificial agent (adjusted odds ratio 2.37; 95% CI 1.26-4.48; P=.007).

Conclusions:

This study offers valuable theoretical and practical insights for the design of AI chatbots in healthcare, emphasizing the important role of message humanness in influencing human perceptions. Future research is warranted to facilitate an understanding of the relationship between chatbot identity, message humanness, and health outcomes.