JMIR Publications

ABSTRACT

Background:

Artificial intelligence healthcare chatbot (AIHC) is gaining widespread adoption worldwide. It is imperative to understand the service quality of AIHC. However, there is limited guidance on how to comprehensively evaluate the service quality of AIHC.

Objective:

This study aimed to develop an index system based on SERVQUAL framework for evaluating service quality of AIHC.

Methods:

An initial indicator pool was compiled through comprehensive literature review and consultations with four experts. These indicators were mapped and categorized into five domains adapted from SERVQUAL framework. The experts were invited form hospital, university and health commission by purposive sampling. The service quality index system was identified using a two-round Delphi, which included a virtual meeting between the two rounds. In the third round, indicator weights within each quality domain and subdomain were determined employing Analytical Hierarchical Process (AHP).

Results:

There were 26 indicators identified in the literature, based on which the two-round Delphi was conducted. A total of 20 experts were invited. The response rates in both rounds of Delphi and the AHP were 100%, and the authoritative coefficients were both >0.7. The final service quality index system for AIHC comprises 5 primary indicators and 17 secondary indicators. There are 3 indicators on assurance, 4 on reliability, 3 on human-like, 4 on tangibility, and 3 on responsiveness. The weights of primary indicators from high to low are: assurance (0.239), reliability (0.237), human-like (0.187), tangibility (0.170), and responsiveness (0.167).

Conclusions:

This study pioneers developing a service quality index system for AIHC adapted from the SERVQUAL framework. The results provide a validated tool for evaluating AIHC performance and offer valuable insights for health service managers and developers to enhance AI-driven medical consultation services.