Encouraging better evaluation design and use through a standardized approach to evaluation planning and implementation - Easy Outcomes

Summary

Communicating a program evaluation plan to high-level stakeholders is difficult because of the complexity of such plans and the limited time stakeholders have to consider them. However it is important that, in spite of this, high-level stakeholders have the opportunity to consider what evaluation is possible and what evaluation they can afford to fund. A visual evaluation plan presented in a standardized format and visualized in specifically designed software provides a new and highly effective way of assisting high-level stakeholders to make decisions about evaluation priorities. The Easy Outcomes approach provides a standardized format for such plans. In addition, such visual evaluation plans can be used not only to make planning decisions about the evaluation but also to control and monitor its implementation. Further, the outcomes models within such evaluation plans can be used for strategic planning, monitoring, economic evaluation, outcomes-focused contracting and other purposes.

Encouraging better evaluation design and use through a standardized approach to evaluation planning and implementation – Easy Outcomes 

Paper presented to the 2008 European Evaluation Society Biennial Conference: Building for the Future: Evaluation in Governance, Development and Progress, Lisbon, 1-3 October 2008. 

Paul Duignan, PhD

Senior Research Fellow

Massey University New Zealand / Parker Duignan Consulting

paul@parkerduignan.com 

Abstract

To maximize evaluation use, high-level stakeholders need to be helped to think about the best way to spend scarce evaluation resources and to focus on how to get evaluation findings that can be used as input into their next round of strategic decision-making and linked to other organizational processes (e.g. monitoring and contracting). Currently, much evaluation planning is undertaken in ways that do not help high-level stakeholders in this way. Long textual evaluation documents are prepared by technical experts in a range of different formats. However, high-level stakeholders often do not have time to read these and executive summaries do not provide an effective mechanism for stakeholders to make, or endorse, important decisions about the optimal evaluation strategy. Easy Outcomes is an approach that attempts to provide a standardized visual evaluation plan to help decision-makers make better resource allocation decisions about evaluation. It also provides a way of linking evaluation results directly back into subsequent rounds of strategic planning. The visual evaluation plan is based on a comprehensive visual outcomes model developed in software specifically designed for the purpose. It provides an approach that allows for all of the usual aspects of evaluation planning but that also links evaluation findings with decision-making about priorities during the strategic planning process. In doing this it allows for evaluation results that need to be brought to the attention of those doing organizational strategic planning to be mapped back onto the same comprehensive outcomes model that has been used in evaluation planning. The same outcomes model, with evaluation findings mapped onto it, is then used as the basis for strategic planning thus ‘closing the loop’ by feeding back evaluative findings about current activities into future strategic planning decision-making. (For more information see http://www.easyoutcomes.org; for an example of a web page version of a visual evaluation plan see http://www.outcomesmodels.org/models/communitycentral.html and for DoView outcomes and evaluation software see http://www.doview.com. 

Key words: evaluation planning, outcomes models, learning organizations 

Introduction

Increasing attention is being paid to the interface between evaluation and decision-making. For a considerable time there has been reflection on the ways in which evaluation findings are used in decision-making (Weiss, 1975; Weiss, Murphy-Graham, & Birkeland, 2005). Methods for improving the uptake of evaluation findings have been promoted under the utilization-focused evaluation approach (Patton, 2008). Suggestions have been made for the mainstreaming of evaluation (Sanders, 2003) and then for repositioning this as evaluation capability building (Duignan, 2003). Lastly, the task has been described as evaluation capacity building within an overall context of positioning evaluation at the heart of organizational and societal learning processes (Preskill, 2008).

There are a range of ways in which these aspirations for evaluation playing a larger role in organizational and societal decision-making can be furthered. These include practical approaches to evaluation focused on how stakeholders will use evaluative information, for instance utilization-focused evaluation (Patton, 2008); approaching evaluation as being about the task of empowering those involved (Fetterman, Kaftarian, & Wandersman, 1996); and suggestions for the use of technology (particularly the internet) and for creating a societal ‘tipping point’ around evaluation (Preskill, 2008). Approaches need to be developed which facilitate the operationalization of these aspirations through concrete and easy to use tools that embed and align evaluation planning, implementation and reporting with organizational and sector decision-making.

Purpose of the Easy Outcomes Approach

This paper focuses on one very practical approach to encouraging the integration of evaluation into decision-making processes – the Easy Outcomes approach1. The Easy Outcomes approach has been used so far on a wide range of topics for evaluation planning, these include: conceptualizing the approach to evaluation of the IMF’s multilateral and bilateral economic surveillance program2; evaluating a new national building regulatory regime3; evaluation of a nation-wide academic research output assessment system4; evaluating the use of a national broadcasting charter system; evaluating a national product stewardship program for encouraging sustainable products; evaluating a national air quality program; evaluating a national official archives system; evaluating a national water quality program; evaluating a national land use program; evaluating a national internet-based community networking platform5. Aspects of the approach have also been used in regard to a variety of programs including: coordinating cross-sector programs for financial and other support for tertiary students; a framework for national gambling outcomes; outcomes for promoting sustainable business practices; national justice sector outcomes; regional public health outcomes; national mental health outcomes; reducing health inequalities in regard to heart health; cancer health promotion outcomes; and supporting students with high-learning needs.

Early versions of the approach were implemented using normal office software and narrative text-based documents. Later versions of the models are being developed in specifically developed software – DoView6 which builds a visual evaluation plan based around the development of a comprehensive outcomes model.7

The Easy Outcomes approach has been designed to ensure that evaluation planning takes into account all of the important considerations regarding evaluation design and methods; that the plan is organized around a set of evaluation questions to allow it to be quickly overviewed by decision-makers; and that the connections between evaluation and other organizational functions (e.g. prioritization, strategic planning and contracting) are made explicit and easy to deal with in a holistic, practical and integrated way.

The detailed purposes for which the Easy Outcomes approach has been designed are for it to: 

  • Provide a practical approach to developing an evaluation plan that makes it easier for those planning evaluations (whether evaluators or non-evaluators) to ensure that they have included all of the important elements needed in any evaluation plan.
  • Set out the evaluation plan in a standard format so that each evaluation is not approached as a novel exercise with different evaluation plans laid out in very different formats. The traditional approach has required decision-makers to first work out how the evaluation plan has been set out before they can get to grips with its contents.
  • 'Hard-wire' into the evaluation planning process options for possible evaluation designs and methods so that evaluation planners with various levels of experience and knowledge will all be assisted to consider the important possibilities for evaluation planning, design and methods.
  • Present the evaluation plan in an accessible visual format so that once decision-makers, evaluation peer reviewers, and stakeholders are familiar with the format they can rapidly overview the essential elements in any evaluation plan set out in the format.
  • Base all evaluation planning around a comprehensive visualized outcomes model of the organization, program or policy being evaluated created according to a specific set of guidelines8 in order to keep evaluation planning focused and organized.
  • Ensure that the standard format does not put any constraints on the possibilities for the type of evaluation approach, philosophy or method being used. 
  • Fully integrate evaluation planning and implementation with all other relevant aspects of organizational life to ensure that evaluation activity is aligned and embedded in the organization (e.g. strategic planning, monitoring, economic evaluation and internal and external contracting for programs and services).
  • Facilitate a strategic approach to evaluation planning so that high-level decision-makers can overview all of the potential evaluation questions and decide where to spend their precious evaluation resources in order to maximize the value of the evaluation to promoting organizational and sector learning.
  • Highlight which evaluation questions are not going to be answered as much as which evaluation questions are going to be answered in the interests of transparency around what information the evaluation is, and is not, going to be able to provide for decision-making.
  • Use a visual outcomes model approach to clarify the current confusion around attribution and accountability associated with contracting for programs and services. This ensures a robust approach to alignment of contracts and programs to high-level outcomes without unrealistic expectations being put on providers about accountability for high-level outcomes. 
  • Leverage the work that goes into the development of evaluation plans by continuing to use the same visualized evaluation plan in controlling the implementation of the evaluation and as the framework for reporting evaluation results. 
  • Use software that makes the sharing of evaluation plans easy so that other evaluation planners can quickly borrow an evaluation plan that has been developed and adapt and amend it for the specifics of the program they are planning to evaluate.


Getting better decision-making about evaluation planning

Key to the integration and embedding of evaluation within organizational and sector processes is providing ways in which high-level stakeholders can quickly overview the key questions it is proposed an evaluation should be asking, the evaluation projects which it is proposed will be undertaken to answer those questions, and the methods which will be used within such evaluation projects. Communicating sufficient information to busy high-level stakeholders (but no more than is necessary) is a significant knowledge management problem. The traditional approach is to use long narrative documents that remain largely unread by high-level stakeholders due to their working under tight time constraints. This approach is supplemented with Powerpoint and verbal presentations aimed at translating what is proposed in an evaluation into small ‘sound bites’ which it is hoped high-level stakeholders will be able to process in order to make decisions about evaluation resource allocation.

The reality is that in many instances high-level stakeholders have no option but to just generally endorse an evaluation plan and accept whatever evaluation design and resource allocation decisions have been made by the technical staff that developed the plan. This has several down-sides. First, high-level stakeholders are shielded from the reality of the trade-offs that need to be made in any evaluation planning around what it is appropriate, feasible and affordable to evaluate. Second, they can continue holding the naïve assumption that the evaluation will be robustly establishing causality in regard to the high-level outcomes of the program when, in fact, the design may be able to draw very little in the way of robust conclusions. Third, they will have missed out on the opportunity to assess whether or not it is a wise use of resources to pursue certain types of evaluation (e.g. outcomes and impact evaluation) when it is unlikely that they will be able to provide robust results, rather than spending resources on more feasible types of evaluation (e.g. formative evaluation to maximize the chances of program success). Fourth, they will continue to see evaluation as something which is not fully integrated into the other organizational processes which are involved (for instance in strategy, monitoring, contracting etc.).

Easy Outcomes attempts to cut through these problem by firstly, visualizing the evaluation plan in a format which can always be dataprojected in a meeting with high-level decision-makers – thus eliminating the need to prepare secondary presentations and a base textual evaluation plan. The function of both forms of documentation is met by the visual evaluation plan. Secondly, by basing evaluation planning around the identification and visualization of a set of evaluation questions which are mapped directly onto a visual outcomes model of the organization, program or policy. This helps clarify exactly what an evaluation is focusing on and reduces the normal confusion which surrounds the same evaluation question being discussed in different verbal formulations. Third, by highlighting the evaluation questions which will not be answered as prominently as the evaluation questions which will be answered, high-level decision-makers will be more aware of exactly what types of information will be produced by a particular evaluation. 

Theory underlying Easy Outcomes

Easy Outcomes is based on a comprehensive understanding of outcomes systems developed within the area of outcomes theory (Duignan, 2005). The term outcomes systems is a generic term for any system which attempts to identify, measure, attribute or hold parties to account for outcomes of any type. Such systems are often known by names such as results measurement, performance management, monitoring, evaluation, strategic priority setting and outcomes identification and tracking. Using this approach as the basis for Easy Outcomes means that, from a conceptual point of view, evaluation is not seen as a process that stands in isolation from the other related processed (e.g. monitoring) that are directed at obtaining information about outcomes.

In this conceptualization there are five types of analysis, evidence and reporting which can be brought bear on the question of whether or not an organization, program or policy is achieving its outcomes. These are:

    Type 1: The logic of how it’s believed lower-level steps will lead to higher-level outcomes. In the Easy Outcomes approach this is called an outcomes model (it is often referred to under different names, such as intervention logic, program logic, logic model, program theory, theory of change, results map, ends-means diagram or strategy map). Such models can be justified by analysis and evidence supporting the links between steps and outcomes in the model. When built, such outcomes models should not be restricted just to steps and outcomes that can be measured or attributed to a particular program.

    Type 2: Tracking any improvements in high-level outcomes. These can be described as indicators and they should not be restricted just to indicators that are attributable to a particular program. Mapping these 'not-necessarily attributable indicators' back onto a comprehensive outcomes model is the most effective way of identifying those steps and outcomes that are currently measurable and those that are not.

    Type 3: Reporting on lower-level indicators which are clearly attributable to a program or intervention. These are often termed 'outputs' and are often used as the basis for accountability of the program or intervention. If attributable output indicators do not reach up to the highest-level of outcomes in a model, the mere monitoring of the program through these indicators will not say anything about attribution of changes in high-level outcomes to the program. In these case, specific evaluation rather than monitoring processes are employed as described in the next point.

    Type 4: Robust outcome/impact evaluation designs proving that the program has changed high-level outcomes. There is a set of seven possible outcome/impact evaluation designs which are specified in Easy Outcomes and which are analyzed for their appropriateness, feasibility and affordability in the case of any particular evaluation.

    Type 5: Non-outcome/impact evaluation to describe (process evaluation) or improve (formative evaluation) the program.

Figure 1 below shows the five types of analysis, evidence and reporting that can be used in regard to any outcomes system and which form the basis for the Easy Outcomes approach.  
 


  Figure 1: Five types of analysis, evidence and reporting underlying Easy Outcomes.


The ten steps used in the Easy Outcomes approach

Using the Easy Outcomes approach consists of working through a set of ten steps. Not all these steps will be used in every case. Some steps may be undertaken initially and then other steps added later in the process. The steps that are marked with (Evaluation) in the list below are ones that are used in a standard evaluation plan. The other steps in the Easy Outcomes process may or may not be used in any particular evaluation. These other steps represent the points at which the Easy Outcomes approach links evaluation in with other organizational processes (e.g. prioritization, strategic planning, evidence-based practice, monitoring, contracting etc.).

The ten Easy Outcomes steps are as follows:

Step 1. Plan your Easy Outcomes work 

1. Plan your Easy Outcomes work, identify who is going to be involved in what way in the process. (Evaluation).

Step 2. Build an outcomes model and check the evidence for it 

2a. Build an outcomes logic model of the program, organization, sector or collaboration. (Evaluation). 

2b. Check the evidence for the model as an accurate picture of the world. (Evaluation).

Step 3. Map your activities & priorities onto the model 

3a. Identify strategic priorities for your next planning period.  (If just doing evaluation and monitoring you will probably not do this step). 

3b. Map current or planned activities onto your model.  Use this to work out where there are gaps between what you are doing and your strategic priorities identified above. (If just doing evaluation and monitoring you will probably not do this step). 

Step 4. Identify indicators that measure outcomes 

4a. Put indicators onto the model. Use this to find out what you are, and are not, currently able to measure.   

4b. Identify indicators attributable to particular players. Ones that everyone will agree have been changed by particular players (you may not have to do this step in every situation).

4c. Identify indicator targets and success criteria. These set the levels for making a judgment as to whether an intervention is successful or not. (Evaluation). 

4d. List any indicator project(s) for improving or developing new indicators. 

Step 5. Identify evaluation questions & evaluation projects 

5a. Put evaluation questions onto the outcomes model. Use this to limit your own, and stakeholders, confusion caused by the same evaluation question being asked using different language. (Evaluation).

5b. List evaluation questions going to be answered. Not all evaluation questions will always be answered in every evaluation. You will review this list in the light of Step 5c immediately below. (Evaluation). 

5c. Assess possible outcome evaluation designs. Identify which (if any) of the seven possible outcome evaluation designs used in Easy Outcomes are appropriate, feasible and affordable in this instance and document your decision. (Evaluation). 

5d. List priority evaluation project(s) for answering the priority evaluation questions. (Evaluation).

5e. Identify evaluation methods for evaluation projects. Select from the list of evaluation methods used in Easy Outcomes. (Evaluation).  

Step 6. Identify possible economic evaluation 

6. Identify possible economic evaluation from the list of types of economic evaluation used in Easy Outcomes. (This may be done as part of an evaluation). 

Step 7. Decide on piloting or full roll-out outcome evaluation 

7. Decide on piloting or full roll-out outcome evaluation. Is outcome evaluation going to be attempted on the full roll-out or just in a pilot phase and only best practice application monitored on full roll-out? (Evaluation). 

Step 8. Identify evaluation management issues 

8. Identify evaluation management issues that need to be dealt with (e.g. consultation with stakeholders, risk management). (Evaluation). 

Step 9. Select outcomes-focused contracting arrangements 

9. Select outcomes-focused contracting arrangements – Easy Outcomes identifies three possible contracting arrangements – contracting for attributable indicators (outputs only); for attributable indicators (outputs) and ‘managing for outcomes’; or for not fully controllable outcomes. Select amongst these and use the visualized outcomes model to identify accountabilities in discussions on contracting or delegation.  

Step 10. Use the outcomes model for reporting back results 

10. Use your model for reporting back. Use many aspects of the Easy Outcomes model you have built for reporting back to decision-makers and stakeholders (e.g. on progress on indicators, on findings from evaluation questions, in contract reporting). (Evaluation).

Conclusion

The Easy Outcomes approach offers the possibility of introducing a standard approach to evaluation planning that is also integrated into other organizational and sector strategic and related processes. Experience with its use so far has shown that: it is possible to construct an evaluation plan around an outcomes model using the Easy Outcomes guidelines; that high-level decision-makers can use Easy Outcomes models to prioritize evaluation resource allocation decision-making; and that the approach does provide comprehensive evaluation plans which can also be used to track implementation.

It provides a concrete framework and practical tool that can operationalize the many aspirations for evaluation to be aligned and embedded within organizations in order to promote organizational and wider societal learning and as a result improve high-level decision-making and policy development.

Resources are available for using Easy Outcomes from http://www.easyoutcomes.org, including guidelines for developing outcomes models; Powerpoints to explain the Easy Outcomes approach to stakeholders; and a comprehensive Easy Outcomes workbook which describes in detail how to do each of the ten Easy Outcomes steps and how to use DoView outcomes and evaluation software to visualize the evaluation plan at each stage of its development. 

[Disclosure: Dr Paul Duignan developed the Easy Outcomes approach and is involved in the development of DoView software.] 
 

    References

    Duignan, P. (2003). Mainstreaming evaluation or building evaluation capability? Three key elements. New Directions for Evaluation, 99, 7-21.

    Fetterman, D. M., Kaftarian, S. J., & Wandersman, A. (1996). Empowerment evaluation. Sage: Thousand Oaks.

    Patton, M. Q. (2008). Utilization-focused evaluation (4th ed.). Thousand Oaks, California.

    Preskill, H. (2008). Evaluation's second act: A spotlight on learning. American Journal of Evaluation, 29(2), 127-138.

    Sanders, J. R. (2003). Mainstreaming evaluation. New Directions for Evaluation, 99, 3-6.

    Weiss, C. H., Murphy-Graham, E., & Birkeland, S. (2005). An alternate route to policy influence: How evaluations affect D.A.R.E. American Journal of Evaluation, 26(1).

    Weiss, C. H. (1975). Evaluation research in the political context. In E. L. Struening & M. Guttentag (Eds.), 1. (pp. 13-26). Beverly Hills: Sage.

    Notes

Comments

©2009 Google