Maintenance Notice

Due to necessary scheduled maintenance, the JMIR Publications website will be unavailable from Wednesday, July 01, 2020 at 8:00 PM to 10:00 PM EST. We apologize in advance for any inconvenience this may cause you.

Who will be affected?

Accepted for/Published in: Journal of Medical Internet Research

Date Submitted: Jul 16, 2019
Date Accepted: Jan 26, 2020

The final, peer-reviewed published version of this preprint can be found here:

Usage Metrics of Web-Based Interventions Evaluated in Randomized Controlled Trials: Systematic Review

Koneska E, Appelbe D, Williamson PR, Dodd S

Usage Metrics of Web-Based Interventions Evaluated in Randomized Controlled Trials: Systematic Review

J Med Internet Res 2020;22(4):e15474

DOI: 10.2196/15474

PMID: 32297870

PMCID: 7193439

Evaluation of web-based interventions: a systematic review of randomised controlled trials

  • Elena Koneska; 
  • Duncan Appelbe; 
  • Paula R Williamson; 
  • Susanna Dodd

ABSTRACT

Background:

The evaluation of web-based interventions (defined as an intervention that can be downloaded or accessed via the Internet through a web browser) in randomised controlled trials has increased over the past two decades. Little is known about how participant engagement with an intervention is measured, reported and analysed in these studies.

Objective:

To review the evaluation of web-based interventions in randomised controlled trials, assessing study characteristics and the methods used to record, and adjust for, intervention usage.

Methods:

A systematic review of the literature was undertaken to identify all published reports of randomised controlled trials that involved a web-based intervention. A random sample of 100 published trials were selected for detailed data extraction. Information on trial characteristics were extracted, including whether web usage data were recorded, and if so, the methods used to gather these data and whether these data were used to inform efficacy analyses.

Results:

A PubMed search identified 812 trials of web-based interventions published up to the end of 2017 and demonstrated a growing trend over time. Ninety of the 100 studies reviewed collected web usage data, but more than half (49, 54%) of these studies did not state the method used for recording web usage. Only four studies attempted to check on the reliability of their web usage data collection methods. Thirty-five (39%) studies reported the patterns or levels of engagement, of which 19 (21%) adjusted for intervention engagement in their outcome analysis, but only two of these used appropriate statistical methods.

Conclusions:

Trialists frequently report a measure of online intervention usage, but do not always report the collection method or provide enough detail on their analysis of the web usage. Appropriate statistical methods to account for the extent of participant engagement are rarely used and are not well reported even in the very few trials which use them. The number of trialists who attempt to check on the reliability of their web usage collection methods is extremely low.


 Citation

Please cite as:

Koneska E, Appelbe D, Williamson PR, Dodd S

Usage Metrics of Web-Based Interventions Evaluated in Randomized Controlled Trials: Systematic Review

J Med Internet Res 2020;22(4):e15474

DOI: 10.2196/15474

PMID: 32297870

PMCID: 7193439

Download PDF


Request queued. Please wait while the file is being generated. It may take some time.

© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.