Full roll-out impact/outcome evaluation versus piloting impact/outcome evaluation plus best practice monitoring

Introduction

It is important to distinguish between two possible overall paradigms for monitoring and evaluation. The first is attempting to undertake impact/outcome evaluation (which attempts to establish that high-level outcomes have been caused by a particular intervention) on the 'full roll-out' of a program. The full roll-out of a program is all of the activity which takes place within a program. For instance, a project which could be being undertaken at 100 sites within a country. The second paradigm is to not attempt to undertake impact/outcome evaluation in the full roll-out of the program, but rather to only attempt to do this in regard to a pilot of the program. In this second approach, the only monitoring and evaluation which would be undertaken for most of the 100 sites would be attempting to ensure that best practice which has been learnt from the pilot is being applied. This may be supplemented with tracking how high-level outcomes are changing in the sector, but without claiming that tracking such indicators (called not-necessarily demonstrably attributable indicators) means that it has been proved that the full program roll-out changed them.

The two paradigms

The evaluation questions being asked in each paradigm

The first paradigm - impact/outcome evaluation on full roll-out - is asking the evaluation question: 'Did this program improve high-level outcomes in the case of the full roll-out of the program?'.

The second paradigm - impact/outcome evaluation only on a pilot and  best practice monitoring regarding full program roll-out, is asking the evaluation questions: 1) 'Did the pilot program improve high-level outcomes?' (an impact/outcome evaluation question); 2) 'What are the details of what the intervention consisted of in the pilot?' (a process evaluation question); and, 3) Is best practice from the pilot being implemented in the full roll-out of the program?' (a formative evaluation question).

The problem with attempting the first paradigm when it is not possible

Attempting the first paradigm approach in cases where it is not appropriate, feasible or affordable to undertake impact/outcome evaluation on the full program roll-out can result in the production of what are called pseudo-outcome evaluations. These are evaluation studies which do not definitively establish whether or not changes in high-level outcomes can be attributed to a program and which are often a waste of time and money.

An example of the second paradigm - normal clinical medical treatment

A good example of the widespread use of the second paradigm is in the area of normal clinical medical treatment. A pragmatic and effective approach to the use of evaluation resources is used in this sector. When a patient visits a physician and they are prescribed a drug treatment, in the normal course of medical practice, there is often no attempt to undertake an impact/outcome evaluation to establish whether any improvement takes place because of the treatment, placebo or some other factor. However, the concept is that the physician will be applying best practice in their decision to give the treatment based on impact/outcome evaluations which have been undertaken in a 'pilot' phase (i.e. in the course of drug trials).

There is currently an irony in the practice of evaluation and evidence-based practice. This is the fact that  clinical medical treatment is viewed as a relatively evidence-focused endeavor - when compared, for instance, with some types of social programs. Such social programs are encouraged to adopt a more evidence-based approach and sometimes clinical medicine is held up as an example (for instance by reference to such comprehensive reviews of effectiveness in medicine as the Cochrane Collaboration). In response the attempt is made to evaluate the effectiveness of such social programs following the example of clinical medicine. However, sometimes the naive attempt is made to evaluate social programs using the first paradigm described in this article - impact/outcome evaluation on full program roll-out, even where this is not appropriate, feasible or affordable. If those designing the evaluation of such social programs clearly differentiated between the two paradigms outlined in this article, they could use the often more appropriate second paradigm - impact/outcome evaluation on piloting and best practice monitoring on full program roll-out. In doing so they would be emulating the normal approach being used in routine clinical medical treatment.

Should measurement of high-level outcomes on full program roll-out be abandoned?

It is important to point out that the second paradigm - impact/outcome evaluation on piloting and best practice monitoring on full program roll-out does not preclude measuring high-level outcomes in the case of full program roll-out. To clarify this point, there are five basic building-blocks of all evaluation and monitoring systems (called outcomes systems). These are: 1) a model of all of the steps which lead up to high-level outcomes (a results or outcomes model); 2) tracking changes in indicators measuring steps and outcomes within the model (without claiming that their mere measurement establishes that they have been caused by a particular intervention; 3) indicators which are demonstrably attributable to a particular intervention; 4) impact/outcome evaluation designs which attempt to establish that an intervention causes high-level outcomes to improve (in the absence of sufficient information from 3 above); and 5) non-impact/outcome evaluation, formative evaluation to improvement program implementation and process evaluation to describe the course and context of a program.

Using this framework, even where building-block 4 (impact/outcome evaluation designs) is not being attempted in the case of full program roll-out, as is the second paradigm, there is no reason why there cannot be monitoring of whether or not high-level outcomes are improving. The key point to understand here is that the mere measurement of such high-level outcomes does not, in itself, establish that they have been improved by the program. However, it is important for strategic purposes to find out (where it is possible to find out) that overall high-level outcomes are improving. Where they are not improving then, obviously, the spotlight will go onto the question of whether or not the original strategy, of which the program is part, is the best strategy in the current circumstances.

Conclusion

There are two overall monitoring and evaluation paradigms regarding impact/outcome evaluation. The first is attempting impact/outcome evaluation on full program roll-out. The second is only attempting it in regard to a piloting phase and then just applying best practice in the case of full program roll-out. Making this distinction can avoid situations where a futile attempt is made to undertake impact/outcome evaluation on full program roll-out. Such futile attempts are reflected in the production of almost useless pseudo-outcome evaluation reports on the full program roll-out. The approach to evaluation planning which distinguishing between these two paradigms feeds into is one which focuses decisions about evaluation planning more as a sector-wide issue rather than one which should just focus on a program as such, see the article on Reframing program evaluation as part of collecting strategic information for sector decision-making for more information on this overall approach. Assessing whether or not the first paradigm is appropriate, feasible and affordable requires being able to assess whether impact/outcome evaluation is appropriate, feasible and affordable. How to do such an assessment with examples is set out in the article Impact evaluation - when it should and should not be done. In the case of the second paradigm, one way of spreading best practice is outlined in the article Best practice representation and dissemination using outcomes models.