JMIR Publications

ABSTRACT

Background:

QUADAS, and more recently QUADAS-2, were developed to aid the evaluation of methodological quality within primary diagnostic accuracy studies. However, its current form, QUADAS-2 does not address the unique considerations raised by artificial intelligence (AI) centred diagnostic systems. The rapid progression of the AI diagnostics field mandates suitable quality assessment tools to determine risk of bias and applicability, and subsequently evaluate translational potential for clinical practice.

Objective:

We aim to develop an AI-specific quality assessment tool (QUADAS-AI), which addresses the specific challenges associated with the appraisal of AI diagnostic accuracy studies. This paper describes the processes and methods that will be used to develop QUADAS-AI.

Methods:

The development of QUADAS-AI can be distilled into three broad stages. Stage 1: A project organisation phase has been undertaken, during which a Project Team and a Steering Committee were established. Following this, the scope of the project was finalised. Stage 2: An item generation process will be completed following: (1) a mapping review, (2) a meta-research study, (3) a scoping survey of international experts, and (4) a patient and public involvement and engagement (PPIE) exercise. A modified Delphi consensus methodology will be carried out to refine the tool, following which the initial QUADAS-AI tool will be drafted. A piloting phase will be carried out to identify components which are considered to be either ambiguous or missing. Stage 3: Specific dissemination strategies will be aimed towards academic, policy, regulatory, industry and public stakeholders respectively.

Results:

Ethical approval to carry out the study, including the Delphi consensus process, has been granted by the Joint Research Compliance Office at Imperial College London (reference number: 21IC6664). QUADAS-AI aims to provide a consensus-derived platform upon which stakeholders may systematically appraise the methodological quality associated with AI diagnostic accuracy studies by the beginning of 2025.

Conclusions:

AI-driven systems comprise an increasingly significant proportion of research in clinical diagnostics. Through this process, QUADAS-AI will aid the evaluation of studies in this domain in order to identify bias and applicability concerns. As such, QUADAS-AI may form a key part of clinical, governmental and regulatory evaluation frameworks for AI diagnostic systems globally.