JMIR Publications

ABSTRACT

Background:

Much has been written on evaluation research methodology and developing and evaluating complex interventions. However there is little guidance on the formative stages prior to setting up an evaluation and how to work with partner organisations that wish to have their provision evaluated to build an evidence base related to their particular local/geographical/cultural context. Research evaluating programmes or interventions rarely discuss the stages prior to evaluation to a) identify suitable projects for evaluation and b) describe the steps required, often with academic institutions working in partnership with external organisations, to set up an evaluation.

Objective:

• To extend work on Evaluability Assessment and pre-evaluation planning by proposing a 8 Step Scoping Framework to be applied prior to Evaluability Assessment to enable the appraisal of multiple programmes in order to identify interventions suitable for evaluation; • To add to literature on Evaluability Assessment and more recent evaluation guidance by describing the processes involved when working with partner organisations.

Methods:

This paper documents the steps required to identify multiple complex interventions suitable for process and outcome evaluation. The steps were developed using an iterative approach, working alongside staff in a local government organisation (Flying Start) with the aim of building an evidence base to demonstrate which interventions improve child outcomes. First and successive drafts of the pre-evaluation framework were presented to Flying Start and council staff, researchers and associated stakeholders over time. The process of identifying suitable programmes for evaluation, and thereby firming up the pre-evaluation steps, was tested using all Flying Start provision.

Results:

The 8 Step Scoping Framework is described using the example of local government organisation Flying Start to illustrate how each step contributes to the ultimate aim of finding projects suitable for process and outcomes evaluation: 1) Formulating overarching key questions which encompass all programmes offered by an organisation; 2) Gaining an in-depth understanding of the work and provision of an organisation and engaging staff; 3) Completion of a data template per project/programme offered; 4) Assessing robustness/validity of data; 5) Decision on projects suitable for evaluation and additional data required; 6) Negotiating with chosen project leads, both within and external to organisation; 7) Development of individual project evaluation protocols; 8) Applying for University and partner organisation ethical approval.

Conclusions:

This paper describes the processes involved in identifying suitable projects for evaluation. It adds to existing literature on assessment of specific programmes suitability for evaluation and guidance on conducting evaluations by setting out the formative steps required to identify suitable programmes from a range of provision. This relates particularly to academic partners and organisations tasked with delivering evidence-based services designed to meet local need. The steps identified have been described in the context of early years provision but can be applied to a range of community-based evaluations, or more generally where an academic partner is working with external stakeholders to identify projects suitable for academic evaluation. Clinical Trial: ISRCTN registry with study ID ISRCTN26468622 http://www.isrctn.com/ISRCTN26468622