JMIR Publications

ABSTRACT

Background:

In recent years, there has been an upwelling of artificial intelligence (AI) studies in the healthcare literature. During this period, there has been an increasing number of proposed standards to evaluate the quality of healthcare AI studies.

Objective:

This rapid umbrella review examines the use of AI quality standards in a sample of healthcare AI systematic review articles published over a 36-month period. The objective of this rapid umbrella review was to examine the use of AI quality standards based on a sample of published healthcare AI systematic reviews.

Methods:

We employed a modified version of the Joanna Briggs Institute umbrella review method. Our rapid approach was informed by Tricco’s practical guide to conducting rapid reviews. Our search was focused on the MEDLINE database supplemented by Google Scholar. The inclusion criteria were English language systematic reviews regardless of review type with mention of AI and health in the abstract published during a 36-month period. For synthesis, we summarized AI quality standards used and issues noted in these reviews drawing on a set of published healthcare AI standards, harmonized the terms used, and offer guidance to improve the quality of future healthcare AI studies.

Results:

We selected 33 review articles published between 2020 and 2022 in our synthesis. The reviews covered a wide range of objectives, topics, settings, designs, and results. Over 60 AI approaches across different domains were identified with varying levels of detail spanning different AI lifecycle stages making comparisons difficult. Healthcare AI quality standards were applied in only 13 of 33 (39.4%) reviews, and in 25 out of a sample of 178 (14.0%) original studies from the reviews examined, mostly to appraise their methodological or reporting quality. Only a handful mentioned the transparency, explainability, trustworthiness, ethics, and privacy aspects. A total of 23 AI quality standard related issues were identified in the reviews. There was a recognized need to standardize the planning, conduct, and reporting of healthcare AI studies, and to address their broader societal, ethical, and regulatory implications.

Conclusions:

Despite the growing number of AI standards to assess the quality of healthcare AI studies, they are seldom applied in practice. With increasing desire to adopt AI in different health topics, domains and settings, practitioners and researchers must stay abreast of and adapt to the evolving landscape of healthcare AI quality standards and apply these standards to improve the quality of their AI studies.