JMIR Publications

ABSTRACT

Background:

AI-driven clinical decision support (CDS) tools offer promising solutions for healthcare delivery by optimising resource allocation, detecting deterioration and enabling early interventions. However, adoption remains limited due to insufficient validation and a lack of transparency. eXplainable AI (XAI) provides users with insights into AI-driven recommendations, but discrepancies in explanations, known as the "Disagreement Problem", can undermine trust and at worst, lead to poor clinical decisions.

Objective:

This study explores the perspectives of clinicians from Australian critical care settings on XAI and the impact of discrepancies in AI-generated explanations on decision-making.

Methods:

Qualitative data were collected using semi-structured interviews with 14 clinical experts, incorporating scenario-based exercises, and analysed using inductive thematic analysis.

Results:

Key factors influencing trust in XAI are identified, and the role of explainability in AI-driven tools are highlighted. Explainability was considered valuable, especially in unfamiliar situations or complex decisions, if the explanations were clear, plausible, and actionable. Discrepancies in explanations generated by different XAI methods are not the primary concern for clinicians, provided the AI’s prediction was accurate and the explanations offered actionable insights aligning with their mental model.

Conclusions:

This study has identified design recommendations and implementation strategies for developing trustworthy, user-centric XAI-supported CDS tools. It also highlights that discrepancy between different explanations is not inherently problematic, provided the explanations are consistent with clinicians’ reasoning. Recommendations made highlight the importance of aligning the design and implementation of AI tools with clinicians’ needs to enhance trust, mitigate risks, and promote successful adoption for improved patient outcomes.