JMIR Publications

ABSTRACT

Background:

Image-based pill verification systems demonstrate high model accuracy. However, their effectiveness in pharmacy practice depends on how pharmacists interact with AI output. The timing of AI advice is one design factor that influences these interactions, yet its impact on pharmacists' moment-to-moment trust dynamics during medication verification requires further investigation.

Objective:

This study aims to investigate how the timing and conditionality of AI assistance shape pharmacists’ trust dynamics during medication verification.

Methods:

Fifty licensed pharmacists completed a simulated medication dispensing task with two AI types: Ex-Ante-Advice (AI advice given concurrently with clinical information) and Ex-Post-Advice (AI advice given after an initial diagnosis). Ex-Post-Advice was further separated into the Involved-Ex-Post and the Not-Involved-Ex-Post conditions. The experiment employed a within-subjects design with varying AI types and AI recognition patterns (Right fill-Correct recognition, Right fill-Incorrect recognition, Wrong fill-Correct recognition, and Wrong fill-Incorrect recognition).

Results:

Trust adjustment magnitude differed significantly across AI assistance conditions. The Involved-Ex-Post condition led to the largest trust adjustment magnitude, followed by Ex-Ante-Advice (P<.001), with the Not-Involved-Ex-Post condition showing the smallest magnitude (P<.001). Analysis by each recognition pattern revealed significant differences in trust adjustment when the right drugs were incorrectly rejected. In this pattern, the Involved-Ex-Post condition led to greater trust adjustment than both Ex-Ante-Advice (P=.0074) and Not-Involved-Ex-Post conditions ($P=.012$). A marginal difference was observed between Ex-Ante-Advice and Not-Involved-Ex-Post condition (P=.052). No significant differences were observed for other recognition patterns.

Conclusions:

Both the timing and conditionality of AI assistance influenced pharmacists' trust dynamics. Disagreement-based AI interventions (Involved-Ex-Post) that incorrectly challenged pharmacists led to a substantial trust decrement, whereas the Not-Involved AI intervention resulted in more stable trust fluctuations. These findings highlight the importance of designing AI systems that align intervention strategies with user expertise and task demands to foster appropriate trust in safety-critical workflows. Clinical Trial: ClinicalTrials.gov (NCT06245044): https://clinicaltrials.gov/search?id=NCT06245044.