JMIR Publications

ABSTRACT

Background:

Evaluating implementation of digital health interventions (DHIs) in practice settings is complex, involving diverse users and multistep processes. Proactive planning can ensure implementation determinants and outcomes are captured for hybrid studies, but operational guidance for DHI studies is limited.

Objective:

We aimed to proactively define, prioritize and operationalize measurement of implementation outcomes and determinants for a DHI hybrid effectiveness-implementation trial. We describe unique advantages and limitations of planning the trial among a large-scale cohort study population and share results of a pre-trial organizational readiness assessment.

Methods:

We planned a cluster randomized, type II hybrid effectiveness-implementation trial testing PositiveLinks, a smartphone application for HIV care, compared to usual care (n=6 sites per arm), among HIV outpatient sites in the DC Cohort Longitudinal HIV Study in Washington, DC. We 1) Defined components of the DHI and associated implementation strategy, 2) Selected implementation science frameworks to accomplish evaluation aims, 3) Mapped framework dimensions, domains, and constructs to implementation strategy steps, 4) Modified or created instruments to collect data for implementation outcome measures and determinants and 5) Developed a compatible implementation science data collection and management plan. Provider baseline surveys administered at intervention sites probed usage of digital tools and assessed provider readiness for implementation with the Organizational Readiness to Implement Change (ORIC) tool.

Results:

Specification of components of the DHI and implementation strategy identified relevant platform usage data and necessary implementer actions, toward planning measurement of program reach and adoption. A priori mapping of implementation strategy steps to the Reach Effectiveness Adoption Implementation Maintenance (RE-AIM) framework identified how data would be collected for each step/corresponding outcome measure. The multi-site cohort study provided infrastructure for prospective, scaled implementation research, with site research assistants (RAs) coordinating implementation. Existing cohort study tools (periodic site assessments, patient consent logs) were adapted for this evaluation to capture representativeness and reach/adoption ‘denominators.’ New survey instruments allowed for framework-guided evaluation of provider adaptations, dose, and fidelity to planned implementation across more participants and multiple timepoints. Some aspects of real-world PositiveLinks implementation were not reflected within the planned hybrid trial (e.g. RAs selected as de facto site implementation leads) or were modified to preserve internal validity of effectiveness measurement (e.g. ‘Community of Practice’). Providers and RAs (n=17) at intervention sites self-reported high baseline usage of digital tools to communicate with patients. Readiness assessment revealed high median total ORIC scores (48; IQR: 45-54), with RAs scoring higher than physicians (52.5 vs 48.0).

Conclusions:

Challenges and opportunities arose in planning the implementation evaluation within a hybrid trial among a cohort population. Prospective hybrid trial planning must balance generalizability of implementation processes to ‘real world’ conditions with rigorous procedures to measure intervention effectiveness. Rapid, scalable tools require further study to enable evaluations within large multi-site hybrid studies. Clinical Trial: NCT04998019