JMIR Publications

ABSTRACT

Background:

The use of artificial intelligence (AI) to generate synthetic images has been increasingly used to overcome the scarcity of large, regulatory-compliant medical images datasets. However, their quality and actual utility in algorithm development across deep learning architectures, medical settings, and image type has been poorly assessed.

Objective:

To revise the use of synthetic images for algorithm development and assess their quality and effect on downstream algorithms trained with these images, according to AI architecture, medical setting, and image type.

Methods:

We conducted a systematic review following PRISMA guidelines. PubMed was searched in January 2024 using terms related to synthetic image generation, diagnostic imaging, and model evaluation, restricted to January 2020-December 2023. Eligible studies evaluated diagnostic, or follow-up applications incorporating synthetic images, reported quantitative measures of image quality or model performance, used human imaging data, and were published as full-text English articles. Studies using synthetic images for non-diagnostic purposes, microscopic/histopathology imaging, and non-primary literature were excluded. Two independent reviewers conducted study selection with discrepancies resolved through group discussion. Data extraction focused on generative architectures, image quality metrics (fidelity, perceptual, Turing-style, distribution-alignment), and downstream model performance outcomes. Risk of bias was assessed using custom criteria based on reporting guidelines for observational studies. Due to insufficient variance reporting across studies, only descriptive analyses (and not pooled analyses) were conducted.

Results:

From 952 articles identified, 94 studies met inclusion criteria. MRI, X-ray, and CT were the predominant imaging modalities (representing the majority of studies), primarily applied in oncology, respiratory diseases, and neurology. GAN-based architectures dominated synthetic image generation (83% of studies). Among 49 distinct quality metrics identified, Turing-style realism metrics were most frequently used, followed by perceptual and fidelity metrics. Seventy-six studies evaluated downstream model performance, revealing that training exclusively on synthetic images often resulted in performance decrements, particularly for segmentation tasks (declining Dice scores). In contrast, augmenting original datasets with synthetic images showed more consistent improvements, especially for classification metrics across most imaging modalities. Risk of bias assessment revealed significant methodological concerns, with only 22% of studies rated as low risk; primary sources of bias included inadequate reporting of sampling strategies, population characteristics, and lack of measures of variance/confidence.

Conclusions:

Synthetic images show promise for addressing dataset limitations in medical AI, but current evidence demonstrates substantial methodological heterogeneity and modest performance improvements that limit clinical translation. The field urgently requires standardized evaluation frameworks and consensus reporting guidelines. Future research should prioritize developing robust, clinically validated frameworks to guide responsible deployment of synthetic image-enhanced AI systems in healthcare. Clinical Trial: PROSPERO (CRD42023369125).