JMIR Publications

ABSTRACT

Background:

The potential of chatbots for screening and monitoring COVID-19 was envisioned since the very outbreak of the disease. Chatbots can help disseminate up-to-date and trustworthy information, promote healthy social behavior and support the provision of healthcare services safely and at scale. In this scenario and in view of its far-reaching post-pandemic impact, it is critically important to evaluate user experience with this kind of application.

Objective:

To evaluate the quality of user experience with a chatbot designed in response to the COVID-19 pandemic by a large telehealth service in Brazil, focusing on an analysis of usability with real users and on an exploration of strengths and shortcomings of the chatbot as revealed in reports by participants in simulated scenarios.

Methods:

We examined a chatbot developed by a multidisciplinary team and used as a component within the workflow of a local public healthcare service. The chatbot had two core functionalities: assisting online screening of COVID-19 symptom severity and providing evidence-based information to the population. We conducted a mixed-methods approach and performed a twofold evaluation of user experience with our chatbot by two methods: (i) a brief five-point Likert-scale questionnaire presented to all users upon concluding their interaction with the bot and responded by 63 people; and (ii) user observation and follow-up interview with 15 volunteer participants.

Results:

Usability assessment with 63 users revealed very good scores for chatbot usefulness (4.57), likelihood of being recommended (4.48), ease of use (4.44) and user satisfaction (4.38). Interviews with 15 volunteers provided insights into strengths and shortcomings in our bot. Comments on positive aspects and problems reported by users were analyzed in terms of recurrent themes. We identified six positive aspects and fifteen issues organized in two main categories: usability of the chatbot and health support offered by it, the former referring to usability of the chatbot and its interactive resources and the latter to the chatbot goal in supporting people during the pandemic through the screening process and education to users through informative content. We found six themes accounting for what people liked most about our chatbot and why they found it useful, three themes pertaining to the usability domain and three regarding health support. Basides positive feedback, our findings identified 25 types of problems producing a negative impact on users, ten of them related to the usability of the chatbot and five related to the health support it provides.

Conclusions:

Our results indicate that users had an overall positive experience with the chatbot and found the health support relevant. Nonetheless, the qualitative evaluation of the chatbot indicated challenges and directions to be pursued in improving, not only our COVID chatbot, but health chatbots in general.