JMIR Publications

ABSTRACT

Background:

As the availability and performance of AI-based clinical decision support (CDS) systems improve, physicians and other care providers poised to be on the front lines will be increasingly tasked with using these tools in patient care and incorporating their outputs into clinical decision-making processes. Vignette studies provide a means to explore emerging hypotheses regarding how context-specific factors, such as clinical risk, the amount of information provided about the AI, and the AI result, may impact physician acceptance and usage of AI-based CDS tools.To best anticipate how such factors influence the decision making of frontline physicians in clinical scenarios involving AI decision support tools, hypothesis-driven research is needed that enables scenario testing before the implementation and deployment of these tools.

Objective:

This study’s objectives are (1) to design an original, online vignette-based survey that features hypothetical scenarios based on emerging or real-world applications of AI-based CDS systems that will vary systematically by features related to clinical risk, the amount of information provided about the AI, and the AI result (2) to test and determine causal effects of specific factors on the judgments and perceptions salient to physicians’ clinical decision-making.

Methods:

U.S.-based physicians with specialties in Family or Internal medicine will be recruited via email and mail (target n=420). Via an online survey, participants will be randomized to a three-part “SMART vignette” detailing a hypothetical clinical scenario involving an AI decision support tool. The SMART vignette design is similar to the sequential multiple assignment randomized trial but adapted to a survey design. Each respondent will be randomly assigned to one of the possible vignette variations of the factors we are testing at each stage, which include: the level of clinical risk, the amount of information provided about the AI, and the certainty of the AI output. Respondents will be given questions regarding their hypothetical decision-making in response to the hypothetical scenarios.

Results:

The study is currently in progress and data collection is anticipated to be completed in 2024.

Conclusions:

The online vignette study will provide information on how contextual factors such as clinical risk, the amount of information provided about an AI tool, and the AI result, influence physicians’ reactions to hypothetical scenarios that are based on emerging applications of AI in frontline healthcare settings. Our newly proposed “SMART vignette” design offers several benefits not afforded by the extensively used traditional vignette design, due to the two aforementioned features. These advantages are: (1) increased validity of analyses targeted at understanding the impact of a factor on the decision outcome, given previous outcomes and other contextual factors, (2) balanced sample sizes across groups. This study will generate a better understanding of physician decision-making within this context.