JMIR Publications

ABSTRACT

Background:

The integration of artificial intelligence (AI) into clinical decision support systems (CDSS) for mechanical ventilation in intensive care units (ICUs) holds great potential. However, the lack of transparency and explainability hinders the adoption of opaque AI models in clinical practice. Explanation user interfaces (XUI), incorporating explainable AI (XAI) algorithms, are considered a key solution to enhance trust and usability.

Objective:

The user-centered design (UCD) of such interfaces continues to be a challenge.This paper presents the first iteration of the design and evaluation phase of an XUI for an AI-based CDSS aimed at optimizing mechanical ventilation in the ICU.

Methods:

A mid-fidelity prototype was designed based on existing guidelines, scientific literature, and insights from previous UCD phases. The prototype was evaluated formatively through two usability walkthroughs (walkthrough 1: N = 4 resident physicians; walkthrough 2: N = 4 ICU nurses), which included guided group discussions and Likert-scale assessments of explanation concepts in terms of understandability, suitability, and visual appeal.

Results:

The XUI was structured into two levels: a first level displaying explanations alongside the CDSS output and a second level offering more detailed explanations for users seeking deeper insight. While both user groups appreciated the first level, physicians found the second level of the XUI useful, whereas ICU nurses found it overly detailed.

Conclusions:

This study underscores the importance of UCD in designing XUIs for CDSS. It highlights the differing information needs of physicians and ICU nurses and emphasizes the value of involving users early in the development of suitable XUIs.