Introduction
Demands are being made in many sectors and countries for program providers to demonstrate that improvements in high-level outcomes be attributed to particular programs or interventions. This is often referred to as impact evaluation, finding out 'what works' and evidence-based practice. In terms of the five building-blocks of outcomes systems framework used in outcomes theory, such demonstration of attribution is the fourth building block (high-level outcome/impact evaluation). Obviously, encouraging the collection of impact evaluation information - where it is appropriate, feasible and affordable to collect - is highly desirable. Such information can be used by funders as a guide to help them direct resources to those programs which are likely to be the most effective. At the same time, as the pressure for establishing 'what works' is being applied, there is also pressure, sometimes from the same funders, for providers to increase collaboration with other programs seeking similar outcomes. This article discusses an issue identified in outcomes theory called the Attribution/Collaboration Trade-off principle which points to the existence of a trade-off between these two different demands.
Trading-off demonstrating attribution against encouraging program collaboration
Outcomes theory identifies that an important trade-off may have to be made in some circumstances between demonstrating attribution to particular programs versus encouraging collaboration between programs. Demonstrating attribution to a particular program requires isolating a particular intervention from other factors in order to identify its specific causal contribution to improving high-level outcomes (often referred to as impacts). In contrast, collaboration requires the close intwining of interventions in time and place so as to achieve synergy between them to make the greatest possible impact on high-level outcomes. There are circumstances where one has the luxury of testing interventions separately and then following this up by testing them operating in combination. The example of therapeutic drugs is such a situation where, it may be possible, in some instances to evaluate the impact of drugs when given separately and then subsequently evaluate their impact when given in cocktails of different types. This of course multiplies the number of impact studies which have to be done which raises the issue of affordability of undertaking a large number of impact studies.
In the real-world setting of many interventions, particularly social programs and international development interventions (for example), there are a number of factors which make such an approach even less appropriate, feasible and, in particular, affordable. In such settings, an unsophisticated expectation or demand that impact evaluation should be the dominant type of evaluation providing information on what works is likely to ultimately create incentives for particular types of programs to be funded. Since, it is generally easier to establish specific attribution to atomistic programs operating in isolation from other programs, outcomes theory would predict that such demands would create incentives to encourage the running of, evaluation of, and ultimately funding of atomistic programs rather than those designed to operated in a collaborative style.
One approach in the face of this potential trade-off is to elevate the attempt at impact assessment up a level from the individual program to the level of collaboration between programs (this is analogous to evaluating a cocktail of drugs in the example given above). However, while useful where this can be done, this in itself does not provide a full solution to the problem because it can introduce complexities in itself. For instance, programs fostering collaboration often collaborate with a patchwork of existing interventions (e.g ongoing state-run education, health or social programs) and other more ad-hoc programs which happen to be occurring at the same time. This can make it hard to establish when the program actually started. It also makes generalizability difficulty because what is being evaluated may be a unique combination of programs in a particular place at a particular time.
Abandoning the assumption that we will always be able to demonstrate attribution
A more nuanced approach to implementing a 'what works' framework, informed by outcomes theory, needs to start by abandoning the assumption that in all (or even most, in some situations) cases it will be appropriate, feasible and affordable to demonstrate attribution of improvements in high-level outcomes to a particular program or intervention if collaboration is being sought at the same time. This position does not diminish the importance and usefulness of impact evaluation where it can be done, and is not a general argument against promoting the use of impact evaluation. It is just a call for being realistic and pragmatic in instances where attempting to demonstrate attribution through impact evaluation may create the unintended negative side-effect of a bias in favor of atomistic programs. Being open to the possibility that demanding impact evaluation in some cases may be counterproductive is consistent with the strategic evaluation approach promoted within outcomes theory. This approach sees collecting evaluation information as a sector-wide task of collecting information relevant to sector decision-making about action rather than a mechanistic insistence that, for instance, impact evaluation always be undertaken. In the strategic evaluation approach, decision-makers take a nuanced approach to identifying priority evaluation questions recognizing the cost of undertaking evaluation and the various trade-offs which are faced between having certainty about measurement and attribution versus being strategic.
An example of the attribution/collaboration trade-off in action
Often those in charge of teams forgo attribution in the interests of getting the job done. They do this because they realize that if they insist on attempting to identify which of the individual team members was responsible for a result which needed the team to achieve it, they will distort the way the team functions. In such a case team members will focus on attempting to have their individual contribution identified, measured, recognized and rewarded. The best way of doing this is to not collaborate where collaboration will tend to obscure their individual contribution. They will tend to act in such a way that they make separate contributions which are easily visible as such. In such situations,to prevent this sort of atomistic behavior, incentives are often allocated as team incentives and explicit incentives may be provided for team collaboration (e.g. ratings by other team members for level of cooperation) rather than incentives for individual contribution (which require the demonstration of attribution to individuals in the team).
Dealing with the attribution/collaboration trade-off issue in 'what works' systems
Nuanced 'what works' systems need to have a mechanism for acknowledging that demonstration of attribution must sometimes be forgone for pragmatic reasons in cases where collaboration is to be encouraged. This can be done by recognizing the programs and types of program where this issue might arise and factoring this into decision-making about what level of demonstration of attribution will be demanded from what type of program in what circumstances.
This issue is related to the issue identified in the Equivalent Ease of Impact Evaluation principle within outcomes theory. This principle points out that where there are major differences in the ease of undertaking impact evaluation (appropriateness, feasibility and affordability) in regard to particular program types, this needs to be taken into account in 'what works' evidence-based practice systems if they are to identify what works in regard to all programs rather that have a bias in favor of those for which it is easy to undertake impact evaluation.
The attribution/collaboration trade-off issue should be dealt with in the same way as the equivalent ease of impact evaluation issue as part of developing a comprehensive evaluation plan for a sector. In such a plan, those programs or program types where the attribution/collaboration trade-off is likely to exist need to be identified.
In practical terms this issue is best dealt with by specifying evaluation questions in considerable detail as part of an overall evaluation plan for a sector. This is in contrast to, for instance, only issuing a generalized call to a sector for more impact evaluation to be undertaken. While it takes coordinated work from the center to develop an overall evaluation plan specifying evaluation questions, this more nuanced approach means that issues such as the attribution/collaboration trade-off can be dealt with at the point where priority evaluation questions to be answered by a sector are identified. How a sector-wide evaluation plan can be developed around a visual outcomes model for a sector as part of adopting a Strategic Evaluation approach is set out here.
Summary
This article has outlined the attribution/collaboration trade-off issue identified in outcomes theory; recommended that it be explicitly recognized and dealt with if 'what works' and evidence-based practice systems are going to avoid the unintended consequence of sometimes discouraging collaboration because of insufficiently nuanced demands for demonstrating attribution; and indicated that this should be done as part of developing an overall evaluation plan for a sector using a strategic evaluation approach.
Appropriate citation of this article
This article should be cited as Duignan, P. (2009) Trade-off between demonstrating attribution and encouraging collaboration. (http://knol.google.com/k/paul-duignan-phd/trade-off-between-demonstration-of/2m7zd68aaz774/45).
[Outcomes Theory Article #222]
Comments
Write New Comment ▼
Write New Comment
Sorry! This knol's owner(s) have blocked you from editing, making suggestions, or commenting here.