Paul Duignan, PhD
Senior Research Fellow Massey University New Zealand / Parker Duignan Consulting
paul@parkerduignan.com
Abstract
Evaluators may see themselves as ideally suited to contribute to governance, development and progress through policy development and related processes. However, regardless of evaluators’ views, the powerful players already involved in policy development (stakeholders, politicians, senior public servants, policy analysts, economists and others) will need to be convinced that the evaluation discipline can add real value. What actual value can evaluation add? Evaluation can already claim expertise in evaluating specific programs and evaluators can also be involved in summaries and meta-analyses of sets of individual evaluations. However, evaluation’s case would be advanced if it could bring to the table a powerful tool that makes policy-making easier. Well-visualized comprehensive ‘outcomes models’ are one specific tool with the potential to play a powerful structuring role at the centre of policy-making. Such models are versions of the logic models, programs logics, intervention logics, programs theories and theories of change that many evaluators use in their daily work. However, for use in higher-level policy-making, outcomes models need to be built according to a specific set of guidelines. These guidelines free models from the constraints evaluators have often worked within when building such models (e.g. only including measurable and attributable outcomes, limiting the number of levels within the model, limiting the model to a single printed page). For such models to serve their purpose in high-level policy-making they need to be visualized in outcomes modeling software which allows them to be used in real-time deliberations by policy-makers. Creating and using such models with policy-making groups is a specialist area of technical expertise. Evaluators, given their experience with logic modeling, together with their other expertise, are well positioned to take a lead role in building and supervising the use of such models within the policy-making environment. To take advantage of this they will need to become adept at constructing the type of outcomes models that are most useful in these settings. (For more information on building and using outcomes models see: www.EasyOutcomes.org , for information on software to visualize such models see: www.DoView.com and for examples of such models see: www.OutcomesModels.org ).
Key words: outcomes models, policy decisions, evaluation
Introduction
There are many players from many disciplines who seek to be involved in the policy making process. Evaluators who want to take their place at the policy-making table will need to do more than just make the claim that they should be represented there. The powerful players already involved in policy development (stakeholders, politicians, senior public servants, policy analysts, economists and others) need to be convinced that evaluation as a discipline can add real value to high-level policy making.
What unique contribution can evaluation as a discipline, and evaluators as individuals, make to the high-level policy-making enterprise? There has been considerable discussion in the literature at the conceptual level about the ways in which evaluation is used in decision-making. These uses include: its direct instrumental use in decisions; symbolic use to justify decisions already made by policy-makers; and conceptual use where future policy is influenced in more subtle ways than straight immediate instrumental use (Weiss, Murphy-Graham, & Birkeland, 2005). This paper focuses at a more concrete than conceptual level by proposing the use of a tool - outcomes models constructed in a specific way - by evaluators. It is argued that evaluators taking this tool to the policy-making table will immediately add value to high-level policy discussion in a highly visible way and help to ensure their continuing involvement in such discussions.
Obviously, it is already accepted that evaluators will contribute to policy making at a technical level through their work in designing, planning, executing and reporting on individual evaluations and summarizing and meta-analyzing groups of evaluations. However, such work is likely to be seen by many policy stakeholders as a lower level technical input into policy-making that provides information on what does and does not work for consideration by higher-level decision-makers. This paper proposes that in addition to this work, evaluators should consider involving themselves directly in the high-level decision-making process by supervising the development and use of well-visualized comprehensive outcomes models as a real-time tool for use by high-level stakeholders in their decision-making processes.
Why, apart from evaluators who desire professional expansionism, should anyone be interested in evaluators being more involved in high-level decision-making? The current vogue for evidence-based policy-making carries within it the possibility that overly simplistic approaches will be adopted that will ultimately reduce the effectiveness of using an evidence-based approach. Evaluators, with good technical knowledge are well positioned to help prevent this by bringing their particular perspective to the decision-making table. At its best, the evaluator’s perspective includes a respect for the importance of the use of evidence in decision-making, combined with realism about the feasibility and cost of collecting robust evidence about causality. It is a perspective that is wary of allowing simplistic performance management claims that the mere measurement of improvements in outcomes while a program is running somehow establishes that it was the program that improved them. As evidence-based policy-making grows in popularity it is particularly important that there are people at the policy-making table who are realistic about what types of evidence collection is feasible and affordable and who understand the subtlety of different types of claims around causality and the implications of these claims for decision-making.
What are outcomes models?
What are outcomes models? Outcomes models are a particular type of causal model that sets out all of the steps that need to be achieved in order to get to higher-level outcomes in a particular policy area. They have many similarities to the wide range of causal models currently built by evaluators in their daily work that have recently been described by Frechtling (2007). However, outcomes models also differ in important ways from some of the types of models currently used by evaluators. Models being used by evaluators at the moment go by names such as: logic models, intervention logics, program logics, program theories, theories of change, ends-means diagrams and (in the private sector) strategy maps. Examples of outcomes models can be found at www.outcomesmodels.org.
These types of models all have conventions about the ways in which they should be laid out. In general, practitioners tend to follow these conventions without always carefully examining exactly why it is that they are following them. Often it is not clear that following the conventions will necessarily produce more useful models.
If, as is argued in this paper, outcomes models can be used by evaluators to assist policy-making, it is essential that the models actually do add value to high-level decision-making. For outcomes models to be a useful tool, it is clear that they need to be structured in particular ways.
Criteria for what constitute outcomes models that are fit-for-purpose for high-level policy making are set out below. Outcomes models built according to these criteria differ in some important ways from the models that are used for individual programs. This difference between models that are being used for cross-program comparisons for policy-making purposes rather than just for individual program evaluation is reflected in the examples in Frechtling's (2007) recent book on logic models. Her examples of logic models for cross-program comparisons are somewhat different in structure from those she gives for single-program models. In particular, and as discussed below, the cross-program model examples do not contain an outputs heading in their structure in contrast to the single-program models which do contain such a heading.
The criteria discussed here have been developed drawing on outcomes theory (Duignan, 2005). Outcomes theory provides a comprehensive theoretical framework that identifies the principles underlying results measurement, performance management, monitoring, evaluation, strategy and outcomes identification and tracking systems of any type. Outcomes theory identifies five features of the steps and outcomes that can potentially be placed in a causal model. Any step or outcome can have one or more of the following features, it can be: influencable by a program, controllable by a program, measurable, attributable to a program (i.e. it can be proved that only one particular program caused it to happen) and accountable (those responsible for a program will be punished or rewarded for it).
Drawing on these five features, criteria for outcomes models that are fit-for-purpose for high-level decision-making are as follows:
- Outcomes models should be ‘world-centric’ not ‘program-centric’. The steps and outcomes in world-centric outcomes models should not just be limited to those that are controllable or attributable to a particular program.
- Outcomes models do not have to only include steps and outcomes that are measurable. This is a major departure from many modeling and outcomes specification conventions that limit steps and outcomes within models just to the measurable. If steps and outcomes in outcomes models are limited to only the currently measurable, they are of little use as a tool for high-level strategic decision-making. This is because they give no sense of how well current measurement systems are capturing what it is believed is strategically important. Limiting models to just measurable steps and outcomes can lead decision-makers to just do the easily measurable rather than the strategically important. Of course, measurement is crucial to outcomes thinking, but it should be dealt with after an outcomes model has been built (e.g. by mapping measurements (indicators) back onto an outcomes model which has been constructed on the basis of theorized causality, rather than being limited to just measurable steps and outcomes).
- Outcomes models should have no artificial limit on their size. If outcomes models are to represent the world for decision-making purposes it is unwise for their size to be limited. The most common mistake in this regard is to attempt to fit a model onto a single printed page. This creates several serious problems. First, in almost all significant policy areas, drawing a comprehensive causal map will require more space than just one page. Second, if steps and outcomes are crammed onto a single page it is unlikely that stakeholders or topic experts will feel that they can amend the model when asked to comment on it. The problem is that they will feel that they cannot add a step or outcome without also suggesting that one be removed to allow their new suggested step to fit in. This situation is made even worse when the model is made even more complex by a nest of line and arrow links.
- Outcomes models need to allow for the possibility that any step or outcome can be causally connected to any other step or outcome in the model. The current standard method of visualizing causal links (a drawn line and arrow) has major disadvantages when it comes to visualizing models that have complex patterns of causality. Often what happens is that causal links between steps and outcomes are left out of a diagram because the person drawing the diagram cannot see how they can fit an additional causal line into the diagram. One often unrecognized consequence of the causal line and arrow problem is that models end up being drawn in a ‘siloed’ manner. In such models the constraining rule is applied that any lower-level step is only allowed to contribute to a single higher-level outcome. This means that the pattern of line and arrow causality can be more easily represented in the visualization of the model. This often also occurs in tabular versions of outcomes models as a table format naturally ends itself to lower-level steps leading to single higher-level outcomes. There needs to be some way of representing causal links between steps and outcomes even in situations where they cannot be visualized as line and arrow links.
- Outcomes models should not be compromised by arbitrary constraints that make them harder to interpret. In a number of types of logic models currently being using by evaluators, there is an arbitrary constraint introduced in regard to the level at which steps should appear within the model. In particular there is sometimes the requirement that models should consist of a single layer of outputs, above them a single layer of intermediate outcomes and then above them again, a single layer of final outcomes. The use of this constraint is reinforced by the unexamined acceptance of the one page limit on models discussed above. From a technical point of view, an output can be regarded as a step in a model which has all of the possible features listed above (i.e. it is influencable, controllable, measurable, attributable and accountable in regard to a particular program). While it is important to identify outputs at a later stage in the process, it is not always appropriate to force an outcomes model to have its physical structure determined by what are and are not outputs. This is because to do so is to structure the model on the basis of measurement and attribution. In contrast to this, an outcomes model should be structured in the best possible way to communicate the hypothesized pattern of causality between lower level steps and higher-level outcomes. Forcing a horizontal level (in models which flow downwards from high-level outcomes at the top) defined by measurement and attribution (i.e. a rigid outputs level) often prevents the clear visualization of causality within a model. For instance to take a health example, some interventions (e.g. the administration of a vaccine for meningococcal disease) may be able to be modeled in terms of a single output (the vaccination), a single intermediate outcome (immunity in individuals), and a single final outcome (reduced morbidity and mortality). However, a more social intervention (e.g. teaching school students to not share drink bottles through increased self-esteem leading to increased interest in healthy behavior) may have many steps that would need to be included in the intermediate outcomes part of the model (e.g. increased self-esteem, increased attention to health-promoting messages from teachers, decreased attention to peer pressure, decreased drink bottle sharing, less transmission of the disease). Forcing both of these interventions into an outcomes model framework that only allows a single outputs step and a single intermediate outcome step means that the single steps at the intermediate outcomes level for these two types of intervention would be conceptually quite different. The first one would consist of a single step, whereas the second one, if a meaningful name for it could be identified, would consist of a significant number of steps somehow compacted into a single step. Such compacting of steps is likely to lead to outcomes models that are hard for stakeholders to interpret. Outcomes models for policy-making should therefore be free from such conventional constraints and allow whatever is needed in terms of the number of levels of steps in different parts of the model to accurately reflect the flow of causality at a sufficiently granular level.
- Outcomes models need to be able to be used in all parts of the decision-making process. In order for them to be able to be used in this way, their visualizations needs to be portable across different media so that they can be used whenever and wherever they need to be used. For example, they should be able to be developed and used in real-time during meetings with high-level stakeholders, printed out in a report, and reproduced on an intranet or the internet. Meeting this criteria requires using appropriate software and laying out an outcomes model in a way that ensures that it is portable. In addition, the model needs to be structured in a way that makes it as accessible as possible to busy decision-makers. A major practical visualization issue in moving away from the one page constraint for outcomes models is that working with an integrated single large model can result in a model that it is difficult for stakeholders to grasp due to its size and complexity.
- One way of dealing with this is to divide the model up into an interconnected set of sub-modules. Each sub-module should contain a set of conceptually related steps making it easier for the viewer to grasp. These sub-modules can be conceptualized as a pragmatic set of ‘slices’ taken through the wider world of causality relating to the topic at hand. For example, for many types of programs, slices can be developed which separately set out the steps and outcomes at a national, regional, organizational and individual level. This modular approach also has the advantage that such slices can then be ‘borrowed’ and amended when one is building another outcomes model even if it is on a different topic because the outcomes have been grouped in ways that may be relevant across diverse topics. For instance, the individual ‘slice’ for a conservation outcomes model may be able to be used to generate ideas when one is building an individual ‘slice’ for a health promotion outcomes model even though the topics are different. Of course, in the visualization software which is used, there needs to be a way of linking steps and outcomes across slices within the model so that the criteria set out above of any step or outcome being able to have a causal connection with any other step or outcome in the model is not violated.
Constructing outcomes models that meet these criteria is a skilled task. Evaluators, with their experience of building similar models, would seem to be well placed to take this on as part of the core skills set. A set of guidelines is available for developing outcomes models that meet these criteria.[1]
The use of such models
Once such models are created according to the criteria set out above, they become powerful tools for undertaking a number of tasks central to high-level policy-making. In particular they can be used for identifying strategic priorities, getting strategic priorities peer reviewed, mapping what evidence exists in regard to the hypothesized links between steps and outcomes, identifying which steps and outcomes currently have indicators associated with them, identifying high level evaluation questions, thinking about what economic evaluation is possible, structuring outcomes-focused contracting discussions and visually reporting the results of monitoring and evaluation. All of these uses of outcomes models are set out in detail within the Easy Outcomes approach2 (Duignan, 2008).
The example to be discussed here is mocked up for illustrative purposes only, its subject is climate change. An outcomes model like this would typically consist of many sub-diagrams, or ‘slices’ that have drill-down hyperlinks between them. Developing such a model would take considerable time. For instance it typically takes at least four half-day workshops to develop a model for a small program, but it may take longer if, as would be the case in regard to a climate change model, the model was larger. Such models are best developed by a small working group that calls in other experts for help in developing particular parts of the model. Detailed suggestions as to how to develop this type of model are set out in Duignan (2008). Outcomes models should be as large as they need to be to represent the world in which policy decisions are being made. The author has worked on some models that have up to twenty ‘slices’ (sub-diagrams). There is no reason why models should not be larger than this if needed. Figure 1 below shows several sub-diagrams (slices) with hyperlink drill-downs between them that allow the representation of more detail of the steps within this part of the model.
from an outcomes model on climate change with ‘hop-to’ drill-down hyperlinks
Figures 2, 3 and 4 below show the slices from Figure 1 in more detail. The following points should be noted about these slices. First, the convention used in drawing them is for high-level outcomes to be at the top of the page. Putting the high-level outcomes at the top emphasizes that these models are for encouraging outcomes orientated thinking. Secondly, line and arrow links have been turned off in these visualizations. Experience has shown that line and arrow links make the visualization so complex that stakeholders cannot work effectively with a model.
Figure 3 shows another level of drill-down beneath the outcome ‘Reduced transport-related carbon emissions’. This is accessed by using the ‘hop-to’ hyperlink in the bottom right-land corner of the outcome box. Figure 3 shows how links between steps can be visualized when using models drawn in DoView. The links reside in the model itself even when they are not represented by line and arrows. These links are shown by the DoView link endpoint icon which can be seen showing the causal connections between the step that is highlighted (‘Sufficiently motivated decision-makers’) and other steps on the slice. The inverted V means that a step results from the highlighted step and two small lines coming out of the top of a step box (representing the base of an arrow) shows that a step causes the step that is highlighted.
In Figure 4, a number of the outcomes on the slice are showing the DoView link endpoint icon. The reason for this is that the model is being used to map projects onto steps and outcomes for strategic decision-making by high-level stakeholders. To do this mapping, a new slice has been prepared which sets out a number of projects in the climate change area (in reality there would be many more of these projects in such a mapping exercise). These projects are shown on the slice set out in Figure 5. Each of the projects have been linked to the outcomes to which they are likely to influence in Figure 4. One of the projects on the Figure 5 slice has been highlighted – ‘Local government collaborative climate change project’. This has resulted in all of the outcomes this project is designed to influence showing the DoView link endpoint icon on the slice in Figure 4.
The numbers that appear with the outcomes in Figure 4 have been put there by the author after counting up the number of projects from the Figure 5 slice that aim to influence each outcome. This is done as set out in Figure 6 which shows how the projects which contribute to a particular outcome can be listed via a right click on that outcome – in this case ‘Increased public awareness’.
Figure 7 shows various other ways in which an outcomes model can be used within the policy and decision-making environment. Indicators have been placed on the model (yellow icons) and evaluation questions have also been placed on the model (green circles with question marks in them). Lastly, the link between ‘Safer urban walking environments’ and ‘Increased use of walking’ has a hyperlink next to it which links out to research evidence for this link located on the internet. There are other ways of using
This paper has discussed the use of outcomes models in the high-level decision-making process. Such models need to be structured according to guidelines that make them fit for use in decision-making. They also need to be visualized in software that allow the models to meet the criteria for outcomes models for decision-making set out earlier in this paper. Once outcomes models have been created they can be used for a range of purposes to help the policy-making process. The conceptual complexity of creating elegant outcomes models for policy making that really add value to high-level decision-makers’ work makes it a task suitable for appropriately skilled and experienced evaluators. This paper has argued that evaluators, where they are not yet using such models to support high-level decision-making, should consider doing so.
[Disclosure: Dr Paul Duignan is involved in the development of DoView software]
Duignan, P. (2008). Easy Outcomes Workbook. http://www.easyoutcomes.org/resources.html
Duignan, P. (2008). Encouraging better evaluation design and use through a standardized approach to evaluation planning and implementation - Easy Outcomes . Paper presented at the Eighth European Evaluation Society Conference, Lisbon, Lisbon.
Frechtling, J. A. (2007). Logic modeling methods in program evaluation. San Francisco: John Wiley & Sons.
Weiss, C. H., Murphy-Graham, E., & Birkeland, S. (2005). An alternate route to policy influence: How evaluations affect D.A.R.E. American Journal of Evaluation, 26(1).
[1] Outcomes model guidelines are available from http://www.easyoutcomes.org/guidelines.html .
[2] Information and resources for using the Easy Outcomes approach are available at http://www.easyoutcomes.org.
Comments
Write New Comment ▼
Write New Comment
Sorry! This knol's owner(s) have blocked you from editing, making suggestions, or commenting here.