Implications of an exclusive focus on impact evaluation in 'what works' evidence-based practice systems

There are ongoing debates about the best methods for undertaking impact evaluation. Outcomes theory currently identifies a set of seven possible types of outcome/impact evaluation design for demonstrating attribution of high-level outcomes. Different stakeholders accept different sets of these designs as being appropriate, feasible and affordable ways of demonstrating attribution. The reason that these designs are grouped is that at least some stakeholders in some situations regard each of them as providing evidence for attribution on which they would then use to make decisions about programs. Some stakeholders only regard a few of these designs as acceptable and some insist that certain of them e.g. true experiments, are the 'gold standard' for impact evaluation. This article assumes that within any outcomes system (any system designed to deal with aspects of outcomes specification, measurement, monitoring, attribution and accountability) there is a notion of which of these designs will be taken as providing demonstration of attribution. This can either be formally stated, or more implicitly indicated by attribution been accepted as being demonstrated when particular types of evidence are provided. For example, evidence-based practice systems such as the Cochrane Collaboration in health care and the Campbell Collaboration in the social policy area, provide explicit inclusion criteria for what type of evidence will be regarded as demonstrating attribution within impact evaluation.

It is an outcomes theory principle (the Equivalent Ease of Impact/Outcome Evaluation Principle) that if outcome/impact [1] evaluation (however stakeholders define it in terms of which of the seven possible outcome/impact designs they will accept as having established attribution) is used as the sole method of determining 'what works' within an outcomes system, and where different types of interventions vary in regard to the ease of undertaking such outcome/impact evaluation, those interventions for which it is easiest to undertake outcome/impact evaluation will tend, over time, to become the interventions supported within the system.

In one area where evidence-based practice is used extensively - the therapeutic drugs area - the issue identified in the above principle is not so pressing because there are usually not huge variations in the ease of undertaking impact evaluations for different types of drugs. The use of outcome/impact evaluation studies in the therapeutic drugs area has lead to major advances in treatment effectiveness. Because of this, the success in having a rigorous focus on outcome/impact evaluation in the therapeutic drugs are is regarded as something as a prototype of how 'what works' systems should be implemented. It is, however, important when implementing 'what works' evidence-based practice systems in other sectors, that there  is a careful assessment of the level of variability in undertaking outcome/impact evaluation for different program types within such sectors.

Of course, even in the therapeutic drugs area, it should be noted that the definition of 'ease of undertaking outcome/impact evaluations' includes the appropriateness, feasibility and affordability of undertaking such evaluation. Affordability needs to be seen in terms of practical affordability within a given sector. Since there is little incentive for drug companies to fund outcome/impact evaluation studies on drugs that they cannot exploit commercially through patenting, it can be expected that there will be a bias in the evidence-base developed within such a system against outcome/impact studies being undertaken on drugs which are less about to be exploited commercially through the use of the patents.

Once one attempts to develop 'what works' comparisons across sectors in which there are widely varying program types, the outcomes theory principle (Equivalent Ease of Outcome/Impact Evaluation) discussed in this article comes more strongly into play. For instance, when comparing individual interventions in controlled settings with community-wide collaborative interventions for which is it much harder to undertake impact evaluations. A simplistic 'what works' system will result in a bias in favor of individual-level interventions simply because of the ease with which these can be subject to outcome/impact evaluation regardless of the potential effectiveness and efficiency of community-wide interventions.

Sometimes what are seen as critiques of current evidence-based practice, such as that implied by the discussion so far in this article, are presented as, or viewed as, global arguments against the whole 'what works' enterprise. This is not the case within outcomes theory where the 'what works' movement is seen as a positive attempt to increase the focus on attributing outcomes. Outcome/impact evaluation designs which are hard to contest because they are robust, are a very potent tool for changing practice which may currently be being determined by professional, commercial or political factors rather than by effectiveness.

The contribution of outcomes theory to developing 'what works' evidence-based practice systems is to highlight the importance of developing sophisticated systems which can cope with variations in the ease of undertaking impact evaluation for different types of programs. If this is not done, the danger is that the 'what works' movement may become discredited because it will simply be the 'what can be demonstrated' movement, and, in addition, there will be no certainty that the most effective and efficient ways of achieving outcomes will be being promoted rather than just those for which impact evaluation is relatively easy. This is exacerbated by the fact that within modern market-based societies there is a bias towards implementing individual level interventions as opposed to community-wide interventions regardless of the potential cost-effectiveness of these two types of intervention. 

The first step in developing a more nuanced approach to implementing a 'what works' system is to, paradoxically, abandon the assumption that in all (or even most, depending on the situation) cases it will be appropriate, feasible and affordable to demonstrate attribution of improvements in high-level outcomes to a particular program or intervention. 

The current debate in the 'what works' area tends to be characterized by two underlying, but opposite, extreme positions. The first position is a simplistic narrowly-defined 'what works' advocacy which tends to implicitly assume that it is possible to establish attribution using outcome/impact evaluation in regard to most important cases. This position usually includes a routine cautionary note saying that demonstrating attribution may be difficult in some instances. However, the underlying logic of the approach means that the assumption needs to be made that demonstrating attribution is possible in most cases because there is usually no mechanism within the system for factoring in ease of impact evaluation. The opposite, extreme position, is one which identifies a whole range of difficulties in demonstrating attribution and implies that the attempt at robustly establishing 'what works' is impossible or nearly impossible.

The approach informed by outcomes theory is a middle way between these two extreme positions which requires an initial formal analysis of the possibility of demonstrating attribution in regard to particular programs and program types. It commences by endorsing the 'what works' aspiration that it would be desirable to be able to robustly demonstrate attribution and that there are methods by which this can be done in at least some instances (using some of the designs set out in the seven outcome/impact designs list). However the approach does not conclude from this, before the fact, that we can base our approach on assuming that demonstrating attribution will be appropriate, feasible and affordable in all important cases. 

There will often be feasibility, and particularly, affordability issues involved in demonstrating attribution which will mean that there will need to be strict prioritization of the outcome/impact evaluation studies which are undertaken. There also needs to be some mechanism for dealing with the fact that the absence of positive impact evaluation findings for those program types where it is more difficult to do impact evaluation has a different significance from the same absence of findings in the case of programs where it is easier to undertake impact evaluation. This level of sophistication is often missing from 'what works' systems. For instance, one can look at a number of evidence summaries in the Cochrane Collaboration collection which are often silent on the issue of the difficulty of undertaking impact evaluation for different types of intervention (e.g. drug interventions versus community health promotion interventions) implicitly assuming that lack of evidence of efficacy means the same in both cases.

Developing a more nuanced 'what works' system which recognizes the outcomes theory principle discussed in this article is entirely possible and holds the promise of avoiding the unintended negative consequences of many current, more naive, what works systems. It involves identifying the following: