Knol will be unavailable during scheduled maintenance starting at Mon, 09 Nov 2009 18:30:00 GMT. We expect the maintenance to be completed at Mon, 09 Nov 2009 20:00:00 GMT.
Version: Baidi441

The Five Building-Blocks of Outcomes Systems

A Topic Article within the Outcomes Theory Knowledge Base

There are five building-blocks which underly all outcomes systems. Outcomes systems are any systems which attempt to deal with specifying, measuring, attributing or holding players to account for changes in outcomes. These systems are called by a range of names such as: results management systems, performance management systems, evaluations etc. The five building-blocks are: 1) an outcomes model/intervention logic; 2) not-necessarily demonstrably attributable indicators; 3) demonstrably attributable indicators; 4) high-level outcome/impact attribution evaluation; and 5) non-outcome (formative and process) evaluation. This article is part of the Outcomes Theory Knowledge Base, an interlinked set of articles detailing the newly developed area of outcomes theory.
Introduction [1]

Discussion regarding outcomes systems (systems which attempt to deal with specifying, measuring, attributing and holding players to account for changes in outcomes of various types) is often confusing because of the diversity of systems used in different sectors, often different names being used for similar systems, and a wide range of different disciplines being involved in such discussions. Outcomes systems are known by names such as: performance management systems, results management systems, monitoring systems, indicator frameworks, program evaluation, evidence-based practice, outcomes-focused contracting and strategic outcomes priority setting processes, amongst others. Prior to the development of outcomes theory, there was no common language for discussing such systems. This article outlines the five basic building-blocks of outcomes systems as identified within outcomes theory. This approach provides a conceptual basis for conducting more coherent discussions regarding the characterization, functioning and improvement of particular outcomes systems.

The Five Building-Blocks

The most important conceptual model underderpinning outcomes theory is that there are five types of evidence which can be offered to show that a program, policy or any intervention 'works'. These are also referred to within outcomes theory as the 'five building-blocks of a comprehensive outcomes system'.

The five building-blocks are set out in Figure 1 below and are:

  • Building-block 1: An outcomes model/intervention logic. An outcomes model sets out the logic of how it is believed lower-level steps undertaken within a program or intervention, will lead to higher-level outcomes. Such outcomes models are often referred to under different names such as intervention logics, program logics, logic models, program theories theories of change, ends-means diagrams or strategy maps. Such models may or may not be justified by analysis and evidence supporting the links between steps and outcomes in the model. In terms of the features of the steps and outcomes which can be put into such models, such outcomes models should not be restricted just to steps and outcomes that can only be measured or attributed to a particular program (measurement and attribution are important but are best dealt with after an outcomes model is drawn). Outcomes models may be presented in various formats, e.g. textual narratives, tables, databases, or in visual models. If outcomes models are to be fit for use within outcomes systems, they should be visualized according to a set of standards to ensure they are well formed and fit for purpose (see Conventions for visualizing outcomes models and Standards for drawing outcomes models).

  • Building-block 2: Not-necessarily demonstrably attributable indicators.These are indicators (measures of steps or outcomes) which track whether or not there has been any improvements in high-level outcomes. These are sometimes described as state or environmental indicators. They should not be restricted just to indicators for which it is able to be demonstrated that they are attributable (able to be proved that they are caused by) to a particular program or intervention. Mapping these not-necessarily demonstrable indicators back onto a comprehensive outcomes model is a powerful way of identifying those steps and outcomes that are currently measurable and those that are not. This is a much better approach than just working with a list of measurable indicators and having no real idea of what is, and is not, currently being measured in terms of outcomes. Tracking trends in not-necessarily attributable indicators is vital for strategic planning in that if not-necessarily demonstrably attributable indicators are improving, that is a much better sign than a situation where they are deteriorating. The first situation does not establish that the change can be attributed to a particular program or intervention merely by their measurement (because they are not-necessarily demonstrably attributable indicators), but the second situation demands an immediate review of the mix or intensity of the programs or interventions that are current being used.

  • Building-block 3: Demonstrably attributable indicators. These are indicators for which it can be clearly demonstrated they are attributable to a particular program or intervention. They tend to be at a lower level within the outcomes model as the closer an indicator is to an intervention within an outcomes model, the easier to demonstrate attribution of changes in the indicator to the model. These are often termed outputs (or more correctly output indicators) and are often used as the basis for accountability of the program or intervention. If demonstrably attributable output indicators do not reach up to the highest level of outcomes in an outcome model, the mere monitoring of the program through indicators will not say anything about attribution of changes in high-level outcomes to the program. In these cases, if one is seeking to prove attribution, specific evaluations, rather than routine monitoring processes need to be employed as described in the next building-block. 

  • Building-block 4: Outcomes/impact evaluation. This is outcome/impact evaluation that attempts to make an attributional claim that it can be proved that a program or intervention has actually changed one or more high-level outcomes in the absence of attributable indicators reaching to the top of an outcomes model. There is a set of seven possible outcome/impact evaluation design types identified in outcome theory that can be used to make such an attributional claim. In the case of any program or intervention, one or more of these designs may or may not be appropriate, feasible and/or affordable. It cannot be assumed before a specific analysis has been undertaken (of the appropriateness, feasibility and affordability of these design types) that one or more of these designs will be appropriate in the case of a particular program or intervention.

  • Building-block 5: Non-outcome/impact (formative and process) evaluation. Non-outcome/impact evaluation describes (process evaluation) or seeks to improve (formative evaluation) a program or intervention. This type of evaluation does not make any high-level attributional claims about being able to prove that a program or intervention has caused high-level outcomes to change. It does however provide rich detailed information about a program or intervention which is useful in its own right in improving the outcomes model, and can also be used to help interpret any outcome/impact evaluation findings coming from Building-block 4 above.

The Five Building-Blocks in Practice

In practice, not every outcomes system will be able to provide evidence and analysis from all of these five building-blocks. If one of them is strong for a particular outcomes system, there may be less need for one or more of the other building-blocks to be emphasized in that particular outcomes system. An important principle of outcomes theory, often violated by high-level stakeholders dealing with outcomes systems, is that it cannot be assumed (or demanded) before the fact that any one of these building-blocks will be able to be provided for a particular outcomes system. In particular, high-level stakeholders often think that the fourth building-block - outcome/impact evaluation can always be undertaken within an outcomes system and that it is appropriate to routinely demand this of outcomes systems (e.g. those funding projects make the demand that outcome/impact evaluation be undertaken in every case). There needs to be an analysis in regard to the particular case to see if outcome/impact evaluation is appropriate, feasible and/or affordable. Another significant mistake often made by stakeholders, and identified by outcomes theory, is to mistakenly assume that the mere measurement of not-necessarily demonstrably attributable indicators establishes attribution for a program or intervention and to hold programs to account for these indicators. This is not the case and systems which attempt to hold programs to account for such indicators ultimately lose their credibility. [2]
Articles which show aspects of how the five building-blocks approach can be used to better conceptualize issues in evaluation and performance management include: Reframing program evaluation as part of collecting strategic information for sector decision-making; Contracting for outcomes; Distinguishing evaluation from other processes (e.g. monitoring, performance management, assessment, quality assurance)

Conclusion

The five building-block framework can be used to analyze any outcomes system to identify gaps and weaknesses in the system and assist in improving it. It can also be used in the design of new outcomes systems to ensure that they are well constructed.


Figure 1: The 5 building-blocks of an outcomes system


Citing this article

Duignan, P. (2008). The five building-blocks of outcomes systems. Outcomes Theory Knowledge Base Article No. 207. (http://knol.google.com/k/paul-duignan-phd/the-five-building-blocks-of-all/2m7zd68aaz774/27).
[Outcomes Theory Article #207]

References

  1. Some of this work was developed when the author was the 2005 New Zealand Fulbright Senior Scholar working at the Urban Institute in Washington D.C
  2. There is a type of contracting in which providers can be held to account for not-necessarily demonstrable (attributable) indicators. However it is only one type of contracting and is often not appropriate in many settings (e.g. many areas of the public service). See the article below.
    Contracting for outcomes

Comments

©2009 Google