What are outcomes systems?

A topic article within the Outcomes Theory Knowledge Base

This article argues that it is useful to view a range of different systems used in various sectors, disciplines and professions as all being types of 'outcomes systems'. Such systems are attempts to deal with specifying, measuring, attributing and holding players to account for changes in outcomes of various types (hence the name outcomes systems). They include results management, performance, monitoring, evaluation, evidence-based practice, contracting and strategy. This article is part of the interlinked set of knols on various topics which make up the Outcomes Theory Knowledge Base.

Introduction  [1]

A common problem of major importance faced in all sectors, disciplines and professions is how outcomes of various types should be specified, measured, attributed, contracted/delegated and who should be held to account for changes in outcomes. Before the development of outcomes theory as a framework for conceptualizing the common principles which underlie all such systems, discussion of these types of systems has taken place using a diverse set of theoretical languages from a range of disciplines (e.g. economics, policy analysis, program evaluation, organization development, Human Resources theory, strategic planning theory etc.). Outcomes theory is an attempt to provide a common conceptual language consisting of a robust set of definitions and principles which can be used to analyze, describe, critique and improve such systems regardless of the sector in which they are being used; the disciplinary language being used to discuss them; or the profession which deals with them in day to day practice. At the heart of outcomes theory is the concept of an outcomes system.

An outcomes system is any system which attempts to deal with specifying, measuring, attributing and holding players to account for changes in outcomes of any type and the steps which lead up to those outcomes. Such system go by a variety of names, such as: results management systems, performance management systems, performance measurement systems, program evaluation, evidence-based practice systems, investment strategies, value-for-money exercises, benchmarking exercises, contracting for outcomes systems, pay for performance systems, etc.

Figure 1: 'Outcomes systems' are operating in areas of activity which have traditionally been seen as somewhat distinct


In order assist thinking about such systems, outcomes theory uses the concept of an outcomes model underlying all outcomes systems of any type. Such an outcomes models, which can be highly complex in the case of some outcomes systems, is conceptualized within outcomes theory as a visualized outcomes structure presented as a vertical visual hierarchy. At the bottom of the hierarchy are the lowest-level actions undertaken by players, these then lead up through a set of steps which are linked by causual links to the highest-level outcomes at the top of the visual model. It is these highest-level outcomes in the outside world which is is hoped a program or intervention will improved.

Needs met by outcomes systems

Outcomes systems conceptualized in this way are systems which attempt to meet a set of needs. The needs that outcomes systems of different types attempt to meet (not all systems attempt to meet all these needs) are set out below:

  1. Identifying and specifying outcomes (the things in the outside world which players are trying to ultimately change through their actions (interventions)) 
  2. Measuring either quantitatively or qualitatively the levels of such outcomes (indicators) 
  3. Setting levels on indicators which parties are attempting to have have occur (targets) 
  4. Comparing the different levels on indicators which different parties have managed to achieve (benchmarks) 
  5. Entering into contracts for, and holding particular parties (providers) to account for, 1-4
  6. Attempting to attribute changes in particular indicators to the actions of particular parties (i.e. proving that it was the actions of a particular party or a particular group of parites which determined a specific change in an indicator) 
  7. Identifying and specifying some, or all, of the relevant lower-level steps in the causal chain which leads from parties' actions right up to high-level outcomes
  8. Identifying priorities for action in regard to lower-level steps within the causal chain in 7
  9. Doing 1-5 in regard to one or more of the lower-level steps within an outcomes model 
  10. Providing analysis regarding the plausibility of the claimed causal link between lower-level steps and other lower-level steps, or lower-level steps and high-level outcomes 
  11. Providing evidence from previous research supporting the claimed causal link between lower-level steps and other lower-level steps, or lower-level steps and high-level outcomes 
  12. Using various research designs (called outcome/impact evaluation designs ) to attempt to attribute changes in high-level outcomes to an intervention. 
  13. In some instances, in the case of 10, attempting to estimate an effect-size for an intervention (the amount of change in a high-level outcome which results from a particular intervention of a particular size) 
  14. Undertaking activity to improve the veracity of the outcomes model (e.g. talking to key informants, undertaking research of various sorts (process evaluation) 
  15. Improving the implementation of interventions through undertaking research and related activities (often called formative or developmental evaluation) 
  16. Attempting to provide cost estimates for undertaking various interventions 
  17. Attempting to provide benefit estimates for various positive high-level outcomes and cost estimates for various negative high-level outcomes 
  18. Combining 13 and 14 into cost benefit and cost-effectiveness analyzes. 

Benefits of thinking in terms of outcomes systems

There is a common set of principles which underly efforts to achieve the eighteen items listed here. It is the task of outcomes theory to clearly articulate these principles and their implications and to provide a robust set of definitions and conceptual model which can improve the way we talk about such systems and facilitate the sharing of knowledge across the many disciplines which currently deal with such systems. If successful, outcomes theory will lead to an effective way of comparing outcomes systems across sectors, communicating the best way such systems should be constructed, and assisting in the improvement of existing systems and the design of better outcomes systems in the future. An example of how this can be done in practice is an analysis of the United Nations Results-Based Management System using the Outcomes Systems Checklist (a checklist of the features of a technically well-constructed outcomes system). 

One specific area where a good understanding of outcomes theory adds value is in regard to misguided attempts to improve outcomes systems which then tend to other problems within the system which were not foreseen because those doing the system redesign do not have an overall cohesive theory of a the basic building blocks of a well constructed outcomes system. This leads to repeated swings in the reform of outcomes systems which tend to emphasis one or another of the basic building-blocks of outcomes systems at the expense of other building blocks. An example of such vacillation is between systems which emphasize measuring high-level outcomes (which ultimately creates frustration because such outcomes cannot be attributed to particular programs); and those which just measure lower-level steps because these can be attributed to particular programs and used to hold them to account (however, these systems, in turn, create frustration because they are seen as being just based on low level outputs. Outcomes theory is designed to help prevent such vacillation which, in the case of large scale public sector systems, for example, can be costly and cause considerable stress for those involved - particularly those at the lower levels within such systems who have to meet a constantly changing set of reporting demands from those above them.

Conclusion

The concept of an outcomes system is a new way of conceptualizing a range of different systems such as performance management, monitoring, results-based, evaluation, evidence-based practice, strategic planning and contracting systems. Outcomes theory lets us conceptualize the principles which determine whether or not such such systems are well constructed and provides specific recommendations for improving existing systems and building better systems in the future. 

Please comment on this article

This article is based on the developing area of outcomes theory which is still in a relatively early stage of development. Please critique any of the argument laid out in this article so that they can be improved through critical examination and reflection.

Citing this article

Duignan, P. (2009). What are outcomes systems? Outcomes Theory Knowledge Base Article No. 216. (http://knol.google.com/k/paul-duignan-phd/what-are-outcomes-systems/2m7zd68aaz774/15).

Links 


[If you are reading this in a PDF or printed copy, the web page version may have been updated].

[Outcomes Theory Article #216]

References

  1. Some of this work was developed when the author was the 2005 New Zealand Fulbright Senior Scholar working at the Urban Institute in Washington D.C

Comments

©2009 Google