Accepted for/Published in: JMIR mHealth and uHealth
Date Submitted: Dec 3, 2018
Date Accepted: Jul 21, 2019
(closed for review but you can still tweet)
Challenges to ‘gold-standard’ evaluations of digital health interventions for young people: Research and implementation lessons from the ARMADILLO randomized controlled trial in Kenya
ABSTRACT
Background:
Evidence on the efficacy of sexual and reproductive health (SRH) communication interventions for youth (aged 15-24), especially from low- and middle-income countries, is lacking. Therefore, the World Health Organization initiated the Adolescent/Youth Reproductive Mobile Access and Delivery Initiative for Love and Life outcomes (ARMADILLO) Study. The ARMADILLO intervention is a free, menu-based, on-demand SMS (text message) platform providing youth-validated SRH information. A randomized controlled trial assessing the effect of the ARMADILLO intervention on SRH-related outcomes was implemented in Kwale County, Kenya.
Objective:
This article describes implementation challenges related to the RCT. These issues, observed during enrolment and early in the intervention period, have implications for digital health researchers and program implementers.
Methods:
This was an open, three-armed, randomized controlled trial. Following completion of a baseline survey, participants were randomized into: the ARMADILLO intervention (Arm 1); a once-a-week contact SMS (Arm 2); or usual care (Arm 3, no intervention). The intervention period lasted seven weeks, after which data collectors administered an endline survey. Two methodological decisions had significant implications for the overall success of the trial’s implementation. As a result, some participants became ‘stuck’ in their progression through the study. The team took a series of measures (reminder SMS, phone calls and eventually correcting the system) so that all participants eventually flowed into the intervention period.
Results:
Three weeks after recruitment began, 660 participants had been randomized; however, 103 (47%) participants in Arm 1 and 140 (69%) in Arm 2 were ‘stuck’ at the language menu. Later, the research team called 231 of these non-engaging participants, and successfully reached 136 to learn reasons for non-engagement. Thirty-two phone numbers were found to be linked with participants who were in violation of the study’s eligibility criteria about phone ownership. Among eligible participants, 30 participants indicated that they had assumed the introductory message was a scam or spam. Twenty-seven participants were confused by some aspect of the system. Eleven were apathetic about engaging. Twenty-four non-engagers experienced some sort of technical issue. All participants eventually started their seven-week study period.
Conclusions:
The ARMADILLO Study’s implementation challenges provide several lessons related to both researching and implementing client-side digital health interventions. Research lessons: 1) have meticulous phone data collection protocols to reduce wrong numbers; 2) train participants on the digital intervention in efficacy assessments, even if it is modelled after existing services; and 3) Client-side digital health interventions have analog discontinuation challenges: factor these into sample size calculations. Implementation lessons: 1) ‘phone ownership’ is a fluid concept; 2) digital health campaigns need to establish a credible presence in a ‘noisy’ digital space; and 3) interest in a service can be sporadic and/or fleeting. Clinical Trial: This trial was retrospectively registered with the ISRCTN Registry and assigned registration number ISRCTN85156148 on 29 May 2018.
Citation
