JMIR Publications

ABSTRACT

Background:

Heart Failure (HF) is a global pandemic having high mortality and rehospitalization rates. Nonadherence to medication is one of the factors contributing to these high rates. Monitoring and motivating HF patients to follow their daily medication intake to improve their health condition is a critical mission for the medical system to deliver long-term care. A conversational agent (i.e., CA) or chatbot has been suggested as a technology that enables this task to become achievable with a large patient population by assisting patients in self-managing their medication routine from home. However, more systematic methods must be used in designing conversational agents for patients' use in Digital Health.

Objective:

This study describes how we have conceived a CA design method by using Design Thinking and the Wizard of Oz in the ongoing development of MARIA- an intelligent heart failure CA into a clinical trial protocol. Our design method's objectives are as follows: i) as an elicitation technique of the end-user needs in the conception of MARIA; ii) as a technique in designing dialogue workflow for MARIA in motivating patients in medication adherence; iii) as a technique for conducting feasibility testing in testing an early prototype of MARIA with end-users for further improvements.

Methods:

We used Design Thinking to elicit the end user's needs in the early conception of MARIA. We integrated the psychoeducational theory - Alderian Therapy, into the ideate stage of Design Thinking for MARIA’s dialogue workflow design. As a testing method for the early conception of MARIA, we design an observational study protocol using the Wizard of Oz method involving end-users. End-users were instructed to interact with MARIA, which a Wizard is operating, operated by pharmacists. The Wizard followed the dialogue workflow for MARIA as a guideline for interacting with the end users. We also perform usability test scoring after each Wizard of Oz experiment to evaluate feasibility and acceptance of using MARIA for medication adherence.

Results:

We have recruited twenty (20) study subjects for the Wizard of Oz protocol. The study subjects interacted with the Wizard an average of 30 interactions with a ratio of 4:1 turn-taking between the Wizard and studied subjects on a topic. The turn-taking indicated that study subjects were engaged in question-answer topics with the Wizard. The topic and speech acts used in the dialogue were aligned with the psychoeducational therapy theory, evidenced by the speech acts annotation, suggestion, support, and applaud. However, having the Wizard play the role of MARIA did indicate gaps in the workflow, such as scenarios in tackling negative responses, appropriate use of emoticons, and the systems feedback mechanism during turn-taking delay. The user interaction with Maria interaction showed that Maria, the median score of human likeness, scored 4.75 out of a total score of 5. However, regarding human likeness, MARIA lacked personality, scoring 3.8. From a patient safety perspective, the role of the Wizard played by the pharmacist was instrumental in determining to what extent a CA communicates on behalf of a healthcare provider on the topic of medication adherence. Originally, we included a workflow in educating about medication side effects but were advised on the implications of automating responses by crawling the Web to share drug side effects search results. Eventually, we removed the workflow for medication side effects.

Conclusions:

Many advances have been made in the use of CA in natural language processing technologies such as the ChatGPT or AWS Lex. Our work aims to leverage upon existing technologies such as ChatGPT or AWS Lex by using a systematic design process in adapting these technologies into healthcare setting that considers patient safety. One of our key results suggests that designing CA benefits from a design process incorporating an observational clinical study protocol for modeling and validating its interaction with users allows us to determine system liability. Incorporating an observational study protocol is an important process if we are to provide CA as a Digital Health solution to patients. The WoZ protocol aided in identifying the technical, content, and workflow improvements of MARIA. In particular, using the Wizard of Oz to design an observational study protocol provides a quick and effective method of determining to what extent the CA will respond on behalf of a healthcare provider and to what extent we automate the sharing of medication information and side effects. The role of the Wizard played by an appropriate expert, in our case by the pharmacists, becomes a critical stakeholder in aiding how the dialogues should be modeled and the various scenarios we must consider from patient safety perspectives. In conclusion, through our design process, we found that the context of providing advice on medication interactions and side effects, are a major concern for pharmacists that, for now, cannot be done on behalf of a chatbot due to the nature of the limited knowledge and evidence-based data pertaining to medication interaction and side effects. Clinical Trial: MARIA_PRO_VER_3_190122