JMIR Publications

ABSTRACT

Background:

Longitudinal senior-care records may support prediction of cognitive and neurological outcomes, but credible evaluation requires transparent handling of test-set reuse, calibration, and incomplete follow-up.

Objective:

To report Gia's original pre-registered evaluation and a post hoc registered comparator addendum on the same held-out population.

Methods:

Gia is a 27-million-parameter transformer trained on discrete clinical event sequences. Data came from one licensed senior-care source containing approximately 12.3 million resident records. Person-disjoint partitions separated development from a test partition containing 468,756 labelled residents and 2,390,627 resident-quarter anchors. Eligibility required at least 20 pre-anchor event tokens and outcome-specific criteria. Five outcomes were evaluated against untuned logistic regression. Discrimination required the lower 95% confidence bound of paired AUROC differences to exceed 0.02; calibration required slope and observed/expected ratio intervals within prespecified bands. Confidence intervals used resident-level bootstrap resampling. Criteria were fixed before testing, with development results in view. The study was internally registered and hash-locked, not publicly registered. The original evaluation and comparator addendum constituted two sealed looks. The addendum compared unchanged Gia predictions with a validation-tuned gradient-boosted model (GBM) and a research anchor baseline, without new pass/fail criteria. No multiplicity adjustment was applied.

Results:

In eligible windows with observed binary outcomes, Gia AUROC (95% CI) was 0.883 (0.880–0.886) for 90-day persistent cognitive transition (8,510 events), 0.828 (0.825–0.831) for 365-day incident dementia (33,557 events), 0.866 (0.863–0.868) for 90-day sustained cognitive decline (15,965 events), 0.782 (0.776–0.788) for 180-day stroke/TIA (4,164 events), and 0.806 (0.788–0.822) for 365-day medication-naive Parkinson's disease (882 events). All five met the original discrimination criterion. Four met calibration criteria; Parkinson's calibration was indeterminate. In the post hoc second look, paired Gia-minus-GBM AUROC differences were 0.0578 (0.0539–0.0619), 0.0375 (0.0349–0.0402), 0.0183 (0.0160–0.0206), 0.0351 (0.0280–0.0424), and 0.1089 (0.0864–0.1292), respectively. All intervals exceeded zero; the original 0.02 margin was not applied to these comparisons. Falls were non-confirmatory replication because prior results were known; hospital return was withheld following a label audit.

Conclusions:

Gia met its original outcome-specific discrimination criteria on five outcomes and calibration criteria on four. The post hoc GBM comparison supported higher retrospective discrimination, with a sustained-decline point estimate of 0.0183 (95% CI 0.0160–0.0206), below 0.02 at the point estimate only. These single-source results support independent-population validation and prospective workflow evaluation; they do not establish benefit from deploying Gia.