Maintenance Notice

Due to necessary scheduled maintenance, the JMIR Publications website will be unavailable from Wednesday, July 01, 2020 at 8:00 PM to 10:00 PM EST. We apologize in advance for any inconvenience this may cause you.

Who will be affected?

Currently accepted at: Journal of Medical Internet Research

Date Submitted: Aug 5, 2026
Date Accepted: Oct 26, 2026

This paper has been accepted and is currently in production.

It will appear shortly on 10.2196/108864

The final accepted version (not copyedited yet) is in this tab.

Calibrated Probabilistic Prediction of Influenza Vaccination Uptake Using Behavioral and Demographic Features from Electronic Health Records

  • Benjamin Kapner; 
  • Tom Mushkat; 
  • Nadav Rappoport

ABSTRACT

Background:

Seasonal influenza vaccination uptake remains low in Israel despite universal health insurance and free vaccine access. Existing vaccination-promotion strategies often use standardized outreach and do not account for heterogeneity in individual vaccination behavior. Prediction models based on routinely collected electronic health record data may enable population segmentation and more targeted allocation of vaccination-promotion resources. However, previous models have focused primarily on discrimination, with limited assessment of calibration and prospective validation across multiple influenza seasons.

Objective:

To develop and prospectively validate a calibrated prediction model for influenza vaccination uptake using behavioral and healthcare-engagement features derived from electronic health records, and to quantify the independent predictive contribution of routine healthcare-engagement features beyond prior vaccination history.

Methods:

We conducted a prediction model development and validation study using data from Clalit Health Services, Israel's largest health care provider. The study included 3 million unique adults aged 18 to 95 years across three influenza seasons: 2022–2023, 2023–2024, and 2024–2025. For each season, features were calculated as of September 1, before the vaccination season, and the outcome was receipt of an influenza vaccine during the subsequent September-to-May season. A Light Gradient Boosting Machine model was developed using 700,000 individuals from the 2022–2023 season and internally evaluated on an independent holdout set of 300,000 individuals. Prospective validation was performed using independent samples of 1 million individuals from each of the 2023–2024 and 2024–2025 seasons. Model performance was assessed using area under the receiver operating characteristic curve (ROC AUC), Brier score, log loss, and expected calibration error. An ablation analysis evaluated model performance after removing influenza and COVID-19 vaccination-history features.

Results:

Influenza vaccination prevalence was 23% in 2022–2023, 20% in 2023–2024, and 22% in 2024–2025. The full model demonstrated high discrimination, with ROC AUC values of 0.946 on the 2022–2023 holdout set, 0.945 on the 2023–2024 prospective validation set, and 0.939 on the 2024–2025 prospective validation set. The corresponding Brier scores were 0.068, 0.067, and 0.069, and expected calibration errors were 0.0033, 0.0194, and 0.0098, respectively. Log loss values were 0.233, 0.229, and 0.239, respectively. Prior influenza vaccination history was the dominant predictor. Removing vaccination-history features reduced ROC AUC to 0.870, 0.872, and 0.860 across the three seasons, respectively, but substantial discrimination remained. Age, nursing-visit frequency, and colorectal cancer screening were among the most influential predictors in the model without vaccination history. Model discrimination and calibration remained consistent across evaluated age, sex, population-sector, and supplemental-insurance subgroups.

Conclusions:

A prediction model based on routinely collected electronic health record data achieved high discrimination and good calibration for influenza vaccination uptake and maintained its performance across two prospectively evaluated influenza seasons. Although prior vaccination history accounted for much of the predictive performance, routine healthcare-engagement features retained substantial independent predictive value. Calibrated subject-level probabilities may support more targeted allocation of vaccination-promotion resources. Prospective intervention studies are needed to determine whether probability-based targeting improves vaccination uptake and whether such strategies can be implemented equitably across populations.


 Citation

Please cite as:

Kapner B, Mushkat T, Rappoport N

Calibrated Probabilistic Prediction of Influenza Vaccination Uptake Using Behavioral and Demographic Features from Electronic Health Records

Journal of Medical Internet Research. 26/10/2026:108864 (forthcoming/in press)

DOI: 10.2196/108864

URL: https://preprints.jmir.org/preprint/108864

Download PDF


Request queued. Please wait while the file is being generated. It may take some time.

© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.