JMIR Publications

ABSTRACT

Background:

Limited data availability and privacy constraints hinder the development of robust survival prediction models for personalized treatment. Synthetic data offers a promising solution, preserving the statistical properties of real clinical data.

Objective:

This study aimed to quantitatively evaluate the clinical applicability of predictive models trained on synthetic data by validating their performance on real-world hospital datasets.

Methods:

We developed and validated colorectal cancer survival prediction models using the National Cancer Data Center (NCDC) synthetic data (30,683 patients from three Korean institutions) for pre-training and real hospital data (2,170 patients from Hwasun Jeonnam University Hospital) for external validation. We evaluated three domain adaptation strategies—domain adaptation, zero-shot, and ensemble—using Extreme Gradient Boosting (XGBoost) and Light Gradient Boosting Machine (LightGBM). In total, 48 model configurations were tested, defined by the combination of algorithm (LightGBM, XGBoost), sampling technique (no-sampling, RUS, SMOTEENN), model type (baseline, domain adaptation, zero-shot, ensemble), and optimization objective (AUPRC, F1). The outcome was seven-year overall survival, evaluated using an Area Under the Precision Recall Curve (AUPRC) and Brier scores. Performance was compared against a hospital-only baseline using absolute values and deltas (ΔAUPRC, ΔBrier). Differences and corresponding 95% confidence intervals were estimated on the held-out test set using 2,000 bootstrap samples.

Results:

Zero-shot application reduced the AUPRC in most settings. In contrast, domain adaptation model improved AUPRC in 9/12 combinations; the best setting (XGBoost +Random Under-Sampling (RUS) +regularized) achieved AUPRC 0.5162 (Δ+0.1026, P=.01). The soft ensemble increased AUPRC in 8/12 combinations, with one statistically significant gain (XGBoost + RUS, Δ+0.0983, P=.04). Across the domain-adaptation runs, the Brier score improved for 24/36 combinations (23 statistically significant), indicating a reliable probability calibration.

Conclusions:

NCDC synthetic data–based models can achieve a performance comparable to that of hospital-trained models using appropriate domain adaptation strategies. This approach offers a viable alternative for developing survival prediction models in healthcare environments with data-sharing constraints. Clinical Trial: Not applicable.