Outcomes systems - checklist for analysis

List of outcomes theory articles

Subscribe to author's e-newsletter

Author's Outcomes Blog
Follow author on Twitter

Introduction

Outcomes systems are any systems which attempt to identify, measure, attribute and hold parties to account for outcomes or the steps which lead to them. Such systems are used in all sectors and go by names such as: performance management systems, results-based systems, managing for outcomes systems, priority-setting systems, strategic planning systems, monitoring systems, evaluation systems, outcomes-focused contracting systems, and evidence-based practice systems. The unique contribution of outcomes theory to analyzing such systems is to provide a common conceptual framework for thinking about them. In contrast, prior to the development of outcomes theory, such systems have often been seen as quite separate and there has not been a general conceptual basis available for critiquing and improving them. 

Using outcomes theory to develop a checklist for analyzing outcomes systems

Using several basic concepts from outcomes theory, any outcomes system can be analyzed to help identify whether or not it suffers from common problems, and recommendations can be made on how to best solve these problems. A checklist has been developed to assist in analyzing any type of outcomes system. An example of the use of the checklist to analyze the United Nations Results-Base Management System is available . The key aspects of outcomes theory which have been used in developing this checklist are: the outcomes systems' building blocks model; key features of steps and outcomes; and conventions for visualizing outcomes models.

1. Outcomes system building blocks model

The first aspect of outcomes theory drawn upon in developing the checklist is the outcomes systems' building blocks model. It sets out the five basic building blocks which need to be in place in any outcomes system if it is to function properly. These are set out in detail in the article on Outcomes system building blocks. They are also shown in Figure 1, and are summarized below:

1. An outcomes model  / intervention logic model which sets out all of the outcomes being sought and the lower-level steps it is believed are necessary to achieve them. 
2. Not-necessarily demonstrably attributable indicators. Measures of steps and outcomes in the outcomes model tracking improvement (but not by their mere measurement necessarily demonstrating that they have been improved by a particular program, organization or intervention).
3. Demonstrably attributable indicators. Measures of steps and outcomes for which it can be proved improvements have been caused by a particular program, organization or intervention.
4. High-level outcome/impact attribution evaluation. Attempts to establish attribution of improvements in high-level outcomes to particular programs, organizations or interventions.
5. Non-outcome (formative and process) evaluation. Non-outcome/impact evaluation to improve the implementation of, or content of, the outcomes model. These are: formative evaluation (for improvement of program implementation) and process evaluation (for describing the course and context of a program).

Figure 1: Outcomes system building blocks

Many of the items in the checklist for analyzing outcomes systems which is set out below is based on assessing whether or not a particular outcomes system has all of these building blocks in place. Systems which do not include the five building blocks often suffer from two types of problems. The first are problems arising because one or more of the building blocks is missing. This means that an important source of information is not available. The second are problems arising when one of the building blocks is being called upon to give information which would be better provided though the use of one or more of the other building blocks. Depending on the circumstances, more, or less, emphasis needs to be put on one or more of the five building blocks.

2. Key features of steps and outcomes

The second aspect of outcomes theory drawn upon in developing the checklist is the key features of steps and outcomes. This is a list of the features of different types of steps and outcomes which can potentially be included within an outcomes model. These features are described in detail in the article on the Key features of steps and outcomes and are summarized below. For a step or outcome to be included within an outcomes model it only needs to have one (or more) of the following features. It can be:

• Relevant – relevant to outcomes it is hoped will be influenced by a program, organization or intervention (this can include what are often referred to as risks or assumptions); and/or,
• Influenceable - able to be influenced by a program or intervention (this is separate from whether or not it can be demonstrated that, in a particular instance, it actually has been influenced by a program, organization or intervention); and/or,
• Controllable - only influenced by one particular program, organization or intervention; and/or, 
• Measurable - able to be measured (this is a separate claim from whether or not its mere measurement has demonstrate attribution to a particular program, organization or intervention); and/or,
• Demonstrably attributable - able to be demonstrated that changes in the step or outcome can be attributed to one particular program, organization or intervention. This is the claim that it can be proved that a particular program, organization or intervention changed a higher-level step or outcome in a particular instance. This is a separate claim from the claim set out above that a higher-level step or outcome is just influenceable by a program, organization or intervention; and/or,
• Accountable (Rewardable or Punishable) - something that a particular program or intervention will be rewarded or punished for.

Often there are constraints put upon what can, and cannot, be put into an outcomes model within particular outcomes systems. For instance, there may be a requirement that an outcomes model be limited to only measurable steps and outcomes, or only to those steps and outcomes which are demonstrably attributable to a particular program, organization or intervention. Such constraints usually creates problems within outcomes systems. The recommended approach within outcomes theory is for outcomes models to be allowed to include any step or outcome which has one (or more) of the above features. For more discussion of this point see the article on Causal models - how to structure, represent and communicate them. 

3. Conventions for visualizing outcomes models which represent the outside world

The third aspect of outcomes theory which has been drawn upon to develop the checklist is criteria for assessing whether or not an outcomes model represented in a particular way is able to truly represent what it is believed is happening in regard to steps and outcomes in the outside world. In particular, outcomes models need to be able to:

• Be as large as is necessary to provide a comprehensive model of all of the steps and outcomes which are relevant to a program (this means breaking out of the traditional one page limitation which applies to may outcomes models (logic models)).
• Allow any set or outcome in the model to be linked to any other step or outcome in the model. In tabular formats for outcomes models in particular, this criteria is often violated.
• Represent measurements (indicators) separately from steps and outcomes (this is to allow non-measurable steps and outcomes within the model).

A more detailed discussion of these criteria is set out in the article on Conventions for visualizing outcomes models.

Checklist for analyzing any outcomes system

Using the basic concepts from the three aspects of outcomes theory which have been summarized above, a checklist has been developed for analyzing any outcomes system. It is set out below. Note that not all of the items in the checklist will be relevant to all outcomes systems, because some outcomes systems may be more limited in the scope than others. The checklist is: 

1. Formal outcomes model as the basis for the system

1.1 Does the outcomes system include a formal outcomes model of some type. Does it include more than just a list of measures without representing the steps and outcomes such measures are attempting to measure separately from the measures?

1.2 Can any step or outcome which has one or more of the following features - relevant, influenceable, measurable, controllable, demonstrably attributable, and/or accountable be included in the outcomes model? Or is it limited in some way - for instance just to the measurable, attributable and/or accountable to a particular party?

1.3 Is the outcomes model represented in a way that allows any step or outcome to potentially cause any other step or outcome so that it can provide the best possible representation of the outside world? Or is the pattern of causality which can be represented limited by problems such as siloization (lower level steps only being allowed to have a causal connection with a single high-level outcome)?

1.4 Can the outcomes model be as large as necessary (e.g. larger than a single printed page)?

1.5 Does the outcomes model attempt to identify where evidence is available regarding causal connections (links) between steps and outcomes?

1.6 Is a distinction made within the system between cases where evidence supporting a causal link between steps and outcomes is missing because: 1) collecting sufficient evidence about a causal link is possible, has been done and support has not been found: 2) collecting sufficient evidence about a causal link is possible, but has not yet been done sufficiently to determine whether the link is supported; or, 3) collecting sufficient evidence about a causal link has not been done because it is difficult for appropriateness, feasibility and/or affordability reasons and hence the absence of supportive evidence does not necessarily mean that there is not a causal link?

1.7 Is there a mechanism for mapping projects or activities onto the outcomes model so as to identify overlaps and gaps in the targeting of activity to assist in prioritizing projects/activities when doing strategic planning?

1.8 Is the outcomes model used for reporting back results from monitoring and evaluation when doing strategic planning?

2.1 Is the difference between not-necessarily demonstrably attributable and demonstrably attributable indicators clearly made within the system? In contrast to attempting to work off a single set of measures (indicators) without making this distinction.

2.2 Are indicators mapped back onto the outcomes mode so that they it can be assessed whether key steps and outcomes are being measured?

2.3 Is there a way of showing which indicators parties will, and will not, be held to account for? Is there provision for parties to identify and/or measure steps or outcomes in some instances, but not be held to account for them? 

3.1 Is the difference between outcome/impact evaluation (which attempts to establish attribution of high-level changes in outcomes) and non-outcome/impact formative/process evaluation (which attempt to improve program implementation or describe the program course and context) made clear within the system so that its stakeholders know where attempts are, and are not, being made to establish high-level attribution through outcome/impact evaluation? 

3.2 Are evaluation questions mapped back onto the outcomes model so that different verbal formulations for the same evaluation question can be controlled and the coverage of evaluation questions across the outcomes model can be assessed?

3.3 Is there a process for prioritizing evaluation questions? Including seeing how they related to sector evaluation priorities?

3.4 Is there a process for assessing the appropriateness, feasibility and affordability of answering high-level outcome/impact evaluation questions being examined and reported on to stakeholders?

3.5 Is a distinction made in the system between whether: 1) high-level outcome/impact evaluation is being attempted on the full roll-out of a program; or 2) just in regard to piloting (and high-level outcome/impact evaluation is not being attempted for full roll-out)? Is there an assessment of the appropriateness, feasibility and affordability of these two different strategies in regard to particular programs?

3.6 Are evaluation questions are not being answered identified as well as evaluation questions for which the attempt is being made to answer them?

4. Has the overall outcomes architecture of the sector, organization or program in which the outcomes system is being used been examined to identify whether there are multiple overlapping outcomes systems operating?

Figure 2: Summary of the Outcomes Systems Checklist

Conclusion

Aspects of outcomes theory have been drawn upon to develop a checklist for analyzing any outcomes system. Where an outcomes system fails a checklist item, the system can be improved by introducing the element into the system which is identified by the checklist item it has failed on. 

 

Please comment on this article

This article is based on the developing area of outcomes theory which is still in a relatively early stage of development. Please critique any of the argument laid out in this article so that they can be improved through critical examination and reflection.

Citing this article

Duignan, P. (2009). Outcomes systems - checklist for analysis. Outcomes Theory Knowledge Base Article No. 240. (http://knol.google.com/k/paul-duignan-phd/outcomes-systems-checklist-for-analysis/2m7zd68aaz774/82).

[If you are reading this in a PDF or printed copy, the web page version may have been updated].

[Outcomes Theory Knowledge Base Article # 240]