JMIR Publications

ABSTRACT

Background:

Artificial intelligence (AI)-based clinical decision support systems are increasingly used in healthcare. Uncertainty-aware AI presents the model’s confidence in its decision alongside its prediction whereas black-box AI only provides a prediction. Little is known about how this type of AI affects healthcare providers’ work performance and reaction time.

Objective:

To determine the effects of black-box and uncertainty-aware AI advice on pharmacist decision-making and reaction time.

Methods:

Thirty licensed pharmacists participated in a crossover, randomized controlled trial. Eligible participants were randomized to either the black-box AI or uncertainty-aware AI condition in a 1:1 manner. Participants completed 100 mock verification tasks with AI help and 100 without AI help. The order of no help and AI help was randomized. Participants were exposed to correct and incorrect prescription fills, where the correct decision was to ‘accept’ or ‘reject’, respectively. AI help provided correct (79%) or incorrect (21%) advice. Reaction times, participant decision, AI advice, and AI help type were recorded for each verification. Likelihood ratio tests (LRT) compared means across the three categories of AI type for each level of AI correctness.

Results:

Participants’ decision-making performance and reaction times differed across the three conditions. Accurate AI recommendations resulted in the rejection of the incorrect drug 96.1% and 91.8% of the time for uncertainty-aware AI and black-box AI respectively, compared to 81.2% without AI help. Correctly dispensed medications were accepted at rates of 99.2% with black-box help, 94.1% with uncertainty-aware AI help, and 94.6% without AI help. Uncertainty-aware AI protected against bad AI advice to approve an incorrectly filled medication compared to black-box AI (83.3% vs 76.7%). When the AI recommended rejecting a correctly filled medication, pharmacists without AI help had a higher rate of correctly accepting the medication (94.6%) compared to uncertainty-aware AI help (86.2%) and black-box AI help (81.2%). Uncertainty-aware AI resulted in shorter reaction times than black-box AI and no AI help except in the scenario where "AI rejects the correct drug". Black-box AI did not lead to reduced reaction times compared to pharmacists acting alone.

Conclusions:

Pharmacists’ performance and reaction times varied by AI type and AI accuracy. Overall, uncertainty-aware AI resulted in faster decision-making and acted as a safeguard against bad AI advice to approve a misfilled medication. Conversely, black-box AI had the longest reaction times, and user performance degraded in the presence of bad AI advice. However, uncertainty-aware AI could result in unnecessary double-checks, but it is preferred over false negative advice, where patients receive the wrong medication. These results highlight the importance of well-designed AI that addresses users’ needs, enhances performance, and avoids overreliance on AI.