JMIR Publications

ABSTRACT

Background:

Artificial intelligence (AI) chatbots, driven by advances in natural language processing (NLP), can analyze and generate human language through computational linguistics and machine learning. Despite the rapid development of large language models, little investigation has been conducted to assess whether AI chatbot-delivered educational conversation can achieve a similar level of efficacy as human-delivered conversation.

Objective:

To evaluate and compare the potential efficacy of human-delivered conversation versus AI chatbot conversation in increasing women’s knowledge and awareness of symptoms and response to heart attack in the United States.

Methods:

This is a secondary analysis of two data sets collected from the AI Chatbot Development Project. Women aged 25 years or older were recruited through flyers and social media. The first dataset contained conversational data where a human researcher engaged in educational conversations with women (Human dataset), whereas the second dataset contained conversational data where an AI chatbot named HeartBot engaged in the same educational conversations with women (HeartBot dataset). Knowledge and awareness of symptoms and response to heart attack were measured at the pre-and post-interaction with either the human or HeartBot. Perceived message effectiveness and conversational quality were measured at the post-survey. Ordinal logistic regression analyses were conducted to explore factors predicting women’s knowledge, adjusting for age, race/ethnicity, interaction group type, education, word count, message effectiveness, and message humanness.

Results:

A total of 171 women (mean age=41 years, SD=12.08) in the Human dataset and 104 women (mean age=46 years, SD=11.86) in the HeartBot dataset completed the baseline survey. Both human-delivered conversations and HeartBot conversation significantly improved participants’ ability to recognize heart attack symptoms (AOR 15.19, 95% CI 8.46-27.25, p=0.000000000000000000075; AOR 7.18, 95% CI 3.59-14.36, p=0.000000025), differentiate between symptoms (AOR 9.44, 95% CI 5.60-15.91, p=0.000000000000000034; AOR 5.44, 95% CI 2.76-10.74, p=0.0000011), call emergency services (AOR 6.87, 95% CI 4.09-11.05, p= 0.000000000000035; AOR 5.74, 95% CI 2.84-11.60, p= 0.0000011), and seek emergency care within 60 minutes of symptom onset (AOR 8.68, 95% CI 4.98-15.15, p=0.000000000000027; AOR 2.86, 95% CI 1.55-5.28, p=0.00078) respectively, even after adjusting for covariates. Comparing the two via interaction tests showed a statistically significant improvement of human-delivered conversation vs. HeartBot conversation for all but the calling an ambulance question (p=0.089).

Conclusions:

Our findings indicate that HeartBot holds promise in increasing heart attack knowledge and awareness among women in a cost-effective manner. Future research should employ rigorous experimental designs, such as randomized controlled trials, and evaluate their effectiveness in improving heart health knowledge and subsequent behavior changes.