JMIR Publications

ABSTRACT

A medical emergency dispatch centre receives a call from a professor informing about a 20-year-old female student showing apparent respiratory distress. After data input, with no known chronic respiratory disease reported, an artificial intelligence (AI) triage system suggests with a probability of 70% that the case is not life-threatening. They send a basic life support ambulance. Eventually, the patient died during transport. The autopsy revealed a pulmonary embolism: she recently started taking oral contraceptives. Clearly, this lack of information affected the AI outcome. Should AI have warned about potentially high uncertainty or asked for more information? Alternatively, without previous embolism, a similar case might have occurred March-2020 as a then-unknown effect of SARS-Cov-2. Health AI should be resilient against uncertain, unprecedented or undesired situations in real-world environments, both in training and during routine clinical use. Data quality and variability issues, fundamental rights assurance or AI regulation are amongst the most relevant challenges. We review requirements and methods for resilient AI in health, and provide a research framework aiming to improve the trustworthiness of next-generation AI-based clinical decision support.