JMIR Publications

ABSTRACT

Background:

Explainable user interfaces (XUIs) for AI-supported clinical decision support systems aim to make model predictions interpretable for clinicians. Two paradigms have emerged: Explanatory XUIs, which deliver static feature-importance summaries and clinical interpretation text and Exploratory XUIs,providing interactive, multi-modal model interrogation. Despite growing interest, empirical evidence comparing both paradigms on decision accuracy, automation bias, and appropriate skepticism in clinicians using a fully automated AI system with deliberately embedded incorrect predictions remains limited.

Objective:

To compare an Explanatory and an Exploratory XUI for AI-supported sepsis risk assessment on decision performance, reliance behavior, user experience, and trust across clinicians.

Methods:

A within-subjects counterbalanced study was conducted with N=17 clinicians (6 physicians, 7 nurses/paramedics, 4 other healthcare professionals; varying intensive care unit (ICU) experience). Participants evaluated three patient cases per XUI condition using SepsisVision, a sepsis risk prediction system, each containing two correct (TP, TN) and one incorrect (FP or FN) prediction. Primary outcomes were post-AI decision accuracy, automation bias, and appropriate skepticism. Secondary outcomes included SUS usability, NASA-TLX workload, task time, trust, explanation satisfaction, and confidence. Within-person XUI comparisons used Wilcoxon signed-rank tests and between-subgroup comparisons (ICU vs. No ICU) used Mann-Whitney U tests, with effect sizes reported as r = |Z|/√N. Binary reliance outcomes were reported descriptively; sepsis experience was examined using Spearman rank correlation. Think-aloud protocols and follow-up discussions were analyzed thematically.

Results:

The Explanatory XUI achieved higher usability (SUS M=85.9 vs. M=71.3, P<.001, r=.898), lower cognitive workload (P=.008, r=.659), and shorter task time (P=.005, r=.687). Post-AI decision accuracy did not differ between conditions (Explanatory M=72.5% vs. Exploratory M=74.5%; P=.756). This stability masked offsetting movements, as accuracy rose on AI-correct cases but fell on AI-incorrect cases (largest drop: Exploratory XUI (FP), -40%), reflecting movement toward the AI regardless of correctness. Automation bias occurred in both conditions (Explanatory 4/17, 23.5%; Exploratory 5/17, 29.4%). Appropriate skepticism was higher with the Explanatory XUI overall (64.7% vs. 47.1%), but moderated by ICU experience. ICU-experienced participants resisted the incorrect recommendation under both interfaces (appropriate skepticism 85.7%), whereas less-experienced participants showed reduced skepticism under the Exploratory XUI (50.0% to 20.0%). Post-AI confidence increased significantly in the Exploratory condition (P=.008, r=.84) but not the Explanatory condition, without accuracy gains in less-experienced users. Sepsis experience correlated strongly with appropriate skepticism in the Exploratory condition (r=+.820, P<.001).

Conclusions:

Average performance was equivalent across XUI types, but the two interfaces diverged by clinical expertise: the Exploratory XUI degraded appropriate skepticism among less-experienced clinicians and raised confidence without improving accuracy, consistent with miscalibrated reliance, while experienced clinicians were unaffected. Because most predictions were correct, this risk was invisible in aggregate accuracy. XUI designs that default to structured, low-burden explanations with progressive access to interactive features are recommended for mixed-expertise settings. Clinical Trial: Dnr 2022-01929-01 Etiksprövningsmyndigheten.