JMIR Publications

ABSTRACT

Background:

A common task by developers and users of synthetic data generation (SDG) methods is to evaluate and compare their utility. Multiple utility metrics have been proposed and used to evaluate synthetic data. However, they have not been validated in general, and not for the SDG method comparison task.

Objective:

Evaluate the ability of common utility metrics to discriminate between SDG methods. The specific workload of interest is using the synthetic data for logistic regression binary prediction models.

Methods:

We evaluated six common utility metrics on 28 different health datasets, and two different SDG methods (a GAN and sequential tree synthesis). These metrics were computed by averaging across 20 synthetic datasets from the same generative model. The metrics were then tested on their ability to select the SDG methods that would have AUROC values in synthetic data most similar to real data on logistic regression binary prediction models. The best performing metrics were then further applied on two separate colon cancer study datasets to narrow them down further in terms of ability to select the best SDG method.

Results:

The utility metrics best able to discriminate between the two SDG methods were the Hellinger distance and the propensity mean squared error (p=0.00006 and both significant at a Bonferroni adjusted alpha level of 0.05). On the validation datasets the propensity mean squared error metric demonstrated greater sensitivity.

Conclusions:

This study has validated a generative model utility metric, the propensity mean squared error, can be used to select among competing SDG methods on the same dataset.