JMIR Publications

ABSTRACT

Background:

Health promotion funders increasingly need evidence of the health and economic impacts of their investments. However, the complex programs vary widely, making HTA-oriented impact evaluation difficult and resource-intensive. A practical screening approach is therefore needed to identify programs with sufficient evidence for further impact evaluation.

Objective:

This study aimed to develop the Health Promotion Categorisation Tool (HPCT), a screening tool to systematically classify health promotion programs by their readiness for HTA-orientated impact evaluation.

Methods:

The HPCT was developed through an iterative, stakeholder-engaged process comprised of three stages: (1) literature and document reviews and stakeholder-informed co-development, (2) multidisciplinary pilot testing and refinement, and (3) pre-liminary inter-rater reliability testing. The tool was refined using qualitative feedback, usability findings, completion time, and agreement between assessors. In Stage 3, two independent evaluators applied the finalized HPCT to 14 ThaiHealth-supported programs selected across different health promotion topics. Overall readiness agreement was assessed using percent agreement and Cohen’s kappa.

Results:

A total of 32 unique participants including health promotion program implementers, evaluators, policy makers, and HTA experts, contributed across the development stages. Stakeholder consultation informed the use of quality-adjusted life years and monetary benefits as impact metrics. The finalized HPCT consisted of three components: Program Information and Program Evaluation Matrix, Outcome Assessment, and a simplified Risk of Bias Assessment. A program was classified as ready when at least one relevant behavioral, clinical, or impact-related outcome was measured both before and after program implementation. Crude agreement during Stage 2 ranged from 75.0% to 91.7%. The mean completion time during pilot testing was 80.3 minutes (SD 31.1). In Stage 3, two evaluators agreed on 13 of 14 programs (92.9%), with a Cohen’s kappa of 0.837. The only disagreement involved a complex report in which relevant outcomes were not clearly reported.

Conclusions:

HPCT offers a transparent and practical approach for assessing the readiness of health promotion programs for HTA-oriented impact evaluation. By providing consistent readiness criteria across diverse programs, HPCT can support more strategic allocation of evaluation resources and strengthen the generation of evidence for health and economic impact.