JMIR Publications

ABSTRACT

Background:

The use of artificial intelligence (AI) to generate synthetic images has been increasingly used to overcome the scarcity of large, regulatory-compliant medical image datasets. However, their quality and actual utility in algorithm development across deep learning architectures, medical settings, and image type has been poorly assessed.

Objective:

To review the use of synthetic images for algorithm development and assess their quality and effect on downstream algorithms trained with these images, according to AI architecture, medical setting, and image type.

Methods:

We conducted a systematic review of studies published between January 2020 and September 2025. PubMed, Scopus, and IEEE Xplore were searched for peer-reviewed studies evaluating synthetic images generated from human medical imaging data. Data were extracted on imaging modalities, generative architectures, image-quality assessment methods, downstream model performance, explainability approaches, reproducibility practices, and risk of bias. Given substantial heterogeneity and incomplete statistical reporting, findings were synthesized descriptively.

Results:

A total of 147 studies were included. GAN-based architectures dominated image generation, although hybrid and diffusion-based approaches became increasingly represented in recent years. Image-quality evaluation was highly heterogeneous, with 49 distinct metrics identified and substantial variability in both human-reader and automated assessments. Models trained exclusively on synthetic images generally performed worse than those trained on real data, with the largest declines observed in segmentation tasks (median Dice change −13.0%). In contrast, augmenting real datasets with synthetic images was associated with modest improvements in classification performance, including precision (+8.5%), recall (+7.6%), F1-score (+6.0%), accuracy (+4.3%), and AUC (+3.2%), whereas gains in segmentation performance were limited (median Dice improvement +1.2%). Explainability evidence was reported in only 54 studies and was rarely validated against expert annotations. Reproducibility reporting was inconsistent, and only 43% of studies were judged to have low overall risk of bias.

Conclusions:

Synthetic images appear more valuable as a complement to real training data than as a replacement for it. Although synthetic augmentation can improve classification performance, benefits are generally modest and less evident for segmentation tasks. The field is characterized by substantial methodological heterogeneity, limited reproducibility, and inconsistent validation practices. Standardized evaluation frameworks and reporting guidelines are needed to support reliable translation of synthetic-image technologies into clinical AI applications. Clinical Trial: Prospero (CRD42023369125).