The Outcomes Theory Knowledge Base is made up of a series of articles on different topics within outcomes theory which are being put up as knol articles. Below is a list of all of these topic articles with a short summary of what is contained within the article. The topic articles which do not yet have links have not yet been posted. In most cases, there is information available about them if you look through the legacy site at www.outcomestheory.org. The legacy outcomes theory site is progressively being transferred to this more accessible Outcomes Theory Knowledge Base structured around Knol articles.
About outcomes theory itself
Aspects of outcomes theory and its applied version Easy Outcomes are being used in a wide range of sectors (e.g. welfare, health, defense, environment, sustainability, natural resource management, culture and heritage, indigenous development) and for a range of purposes including: identifying outcomes, monitoring, reporting, evaluation, prioritization and contracting within strategic planning, performance management, program evaluation, evidence-based practice and contract management.
List of references for use in citing outcomes theory.
A summary list of key outcome theory related resources for working with outcomes, results management, evaluation, performance management, outcomes-focused contracting and evidence-based practice.
Lay summary of Outcomes Theory (not yet posted)
Provides an accessible lay summary of outcomes theory, what value it can add to your thinking about outcomes, results and performance management.
This article is an introduction to outcomes theory which describes the different parts of the approach and what it can be used for.Definitions used in outcomes theory
A comprehensive set of definitions of terms used within outcomes theory and terms used in outcomes systems and their relationship to outcomes theory terms.
Examples showing where Outcomes Theory can add value (not yet posted)
Whether or not outcome theory adds value in practice depends on its ability to quickly cut through current confused discussion regarding aspects of outcomes systems by providing a few key conceptual frameworks which enable outcomes systems to be better understood, discussed and improved. This article provides a set of short stories (vignettes) and shows how insights from outcomes theory can quickly add value to discussion regarding outcomes system.
An outcomes system is any system which attempts to deal with specifying, measuring, attributing and holding players to account for changes in outcomes of any type. Such system go by a variety of names, such as: results management, performance management, performance measurement, evaluation, evidence-based practice, investment strategies, value-for-money exercises, benchmarking, contracting, strategy etc.
Simplifying terms used when working with outcomes
There is a great deal of confusion regarding terms which are used when working with outcomes. Based on the insights of outcomes theory, this article argues that with four relatively simple terms, and heavy use of visual outcomes models, most stakeholder needs in regard to working with outcomes can be met.
There is a great deal of confusion regarding terms which are used when working with outcomes. Based on the insights of outcomes theory, this article argues that with four relatively simple terms, and heavy use of visual outcomes models, most stakeholder needs in regard to working with outcomes can be met.
Five basic building-blocks of all outcomes systems are identified within outcomes theory. This article describes each of the five. They are: an outcomes model; not-necessarily attributable indicators (measures of steps and outcomes in the outcomes model; attributable indicators (ones which it can be proved a particular player improved); outcome/impact evaluation which attempts to prove that a program or intervention caused high-level outcomes to improve; other non-outcome/impact evaluation which focuses on improving or describing a program without making a claim about high-level outcome attribution (called formative and process evaluation).
Having a common outcomes model underpinning multiple organizational activities
The concept of an 'outcomes model' can be seen to underly a wide range of different types of organizational activity - strategic planning, prioritization, performance management, accountability, evaluation, value-for-money exercises, line-by-line reviews, evidence-based practice, research and evaluation planning, outcomes-focused contracting and others. These different types of organizational activity have, to varying degrees, been seen as closely or only distantly related. An appropriately structured 'outcomes model' - a visual representation of all of the high-level outcomes and steps which are needed to achieve them - can serve as a common basis for all of these organizational activities. This approach has the potential to both increase the outcomes credibility of the organization while at the same time providing gains in efficiency through work which was previously duplicated in different formats under different organizational activities being done once and then leveraged off many times for a range of different organizational purposes.
Discussing evaluation is sometimes made difficult by there being a range of different terms for different 'aspects' of evaluation. In some discussions a term attempting to describe one aspect of evaluation is inappropriately presented in opposition to another term attempting to describe a quite different aspect of evaluation. A set of five different aspects of evaluation with corresponding terms can be identified. These are: evaluation approaches (broad, often philosophical, orientations to evaluation); evaluation types (the purpose of an evaluation); evaluation methods (evaluation data collection methods); evaluation information analysis techniques (data analysis and other techniques); and evaluation designs (the way that methods and techniques are sequenced in the case of any particular evaluation). Using this framework can help reduce evaluation discussion in which people 'talk past each other' because terms related to different aspects of evaluation are being inappropriately set up on opposition to each other.
Three 'types' of evaluation can be identified. These are based on the purpose for which an evaluation is being undertaken. The first is formative/developmental evaluation which is focused on ensuring that the implementation of a program is optimized for success; the second is process evaluation which focuses on describing the course and context of a program; and the last is outcome/impact evaluation which attempts to attribute changes in high-level outcomes (intended and unintended, positive and negative) to a particular program or intervention.
What are outcomes models (program logic models)?
Outcomes models (also know as logic models, program logics, program theories, theories of change, outcomes hierarchies, strategy maps, ends-means diagrams etc) are widely used in outcomes and evaluation work. They are the first of the five building-blocks of any outcomes system. There have traditionally been a range of unexamined conventions about how such models should be drawn. 'Full' models which are not limited just to measurable and demonstrable (attributable) steps and outcomes have advantages over more constrained outcomes models. Visualized versions of such models can be used for a range of purposes in strategic thinking, monitoring, evaluation and contracting.
There are various ways of representing causal models, this article concludes that: 1) causal models are best represented as visual models (backed up, where possible, with mathematically-based models); 2) to provide the richest possible model of causality in the real world, there should be the least number of constraints on which causal elements are allowed within a causal model; 3) any element within a causal model should be able to have a causal connection to any other element within a model; and, 4) the 'type' of causal element (as defined by its features) within a causal model needs to be able to be visualized, but this should not conflict with other aspects of the visualization (e.g. showing the upward flow of causality).
An outcomes structure is an information system which brings together information from the five building-blocks of performance management and outcomes systems. The most efficient way to work with such structures is to construct them against a visual outcomes models. This article looks at the advantages of doing this.
Conventions for visualizing outcomes models (program logic models)
Traditionally outcomes models (also known as logic models, program logics, program theories, theories of change, outcomes hierarchies, strategy maps, ends-means diagrams etc) have been visualized using a range of conventions (e.g. top to bottom, left to right on a page). A convention for visualizing outcomes models which can be used across strategic prioritization, program planning and implementation, monitoring, evaluation and contracting is described.
There are six features of the steps and outcomes which can be put into an outcomes model. These are that they can be - relevant, influencable, controllable, demonstrably attributable (able to be proved that they are caused by a particular player) and/or accountable.
The way an outcomes model (intervention logic, ends-means diagram, program theory, logic model, strategy map etc.) is drawn determines its usefulness for a range of purposes. This set of outcomes theory standards sets out how outcomes models should be drawn if they are to provide the most value as an underlying way of representing the link between programs/interventions and higher-level outcomes within outcomes systems.
Indicators - why they should always be mapped onto a visual outcomes model
Indicator lists are usually presented in a tabular format. There are major advantages to always presenting them against the underlying visual outcomes model. This lets you identify which steps and outcomes are currently not being measured and encourages you to attempt to measure them. Even in cases where some steps and outcomes cannot be measured - because of feasibility and affordability - always presenting indicators against the underlying visual outcomes model means that decision-makers are always aware of the limitations of the measurement information they are using to make strategic decisions.
Evaluation questions - why they should be mapped back onto a visual outcomes model
Indicator lists are usually presented in a tabular format. There are major advantages to always presenting them against the underlying visual outcomes model. This lets you identify which steps and outcomes are currently not being measured and encourages you to attempt to measure them. Even in cases where some steps and outcomes cannot be measured - because of feasibility and affordability - always presenting indicators against the underlying visual outcomes model means that decision-makers are always aware of the limitations of the measurement information they are using to make strategic decisions.
A good way of structuring any program evaluation is to initially identify the set of evaluation questions which it is hoped the evaluation will answer. However, in practice, getting evaluation teams and stakeholders to think in terms of evaluation questions can become complex if the questions are not clearly differentiated from each other. A useful way of specifying evaluation questions, and preventing alternative verbal different formulations of the same evaluation questions appearing within an evaluation plan, is to always work with evaluation questions mapped onto a visual outcomes model (a type of program logic model) when developing or working with evaluation questions as part of program evaluation planning.
There are three claims which can be made regarding outcomes model (or sub-parts of outcomes models). These are: 1) that they represent how it is thought that a program will work; 2) that they represent how it is thought that a program will work and that this is supported by evidence from other programs; and 3) that they represent what has been demonstrated to be occurring in the case of a particular program. These claims can be used to mount arguments that stakeholders should have confidence that a program is improving high-level outcomes. These arguments can be positioned roughly at three levels along a continuum of increasing confidence that a program is causing improvements in high-level outcomes.
There are seven possible outcome/impact evaluation designs identified within outcome theory. This article describes each of these designs.Impact evaluation - when it should, and should not, be used
It is often assumed that impact evaluation (sometimes also known as outcome evaluation, or in outcomes theory known as high-level outcome/impact attribution evaluation) should always be attempted for any program, organization or intervention. Impact evaluation is the attempt to prove that changes in high-level outcomes can be attributed to a particular intervention. In contrast, outcomes theory suggests that the appropriateness, feasibility and affordability of doing impact evaluation for any intervention should always be assessed. It is often better to save precious evaluation resources for use only on selected high priority impact evaluations or for non-impact evaluation (e.g. formative evaluation). Attempting impact evaluation where it is not appropriate, feasible or affordable can lead to pseudo-impact evaluation which appear to be impact evaluations, but do not provide robust information sufficient to satisfy key stakeholders that attribution has actually been established.
Full roll-out impact/outcome evaluation versus piloting impact/outcome evaluation plus best practice monitoring
There are two different overall impact/outcome evaluation and monitoring paradigms which can be used. The first is to attempt to undertake impact/outcome on the 'full roll-out' of a program. The second is to just attempt impact/outcome evaluation on a pilot of a program and not attempt it on the full roll-out of the program. In the second case all that is done in regard to full roll-out of the program is monitoring that best practice is being applied. It is very important to distinguish between which of these two possible paradigms is going to be used in an evaluation. Attempting the first paradigm when it is not appropriate, feasible or affordable can lead to producing almost useless pseudo-outcome evaluation reports.
There is a set of commonly used method and types of analysis for collecting data for use in examining the steps and the outcomes they contribute to in outcomes systems. These methods are used to find out more about outcomes models and trends in steps and outcomes within such models.
Economic analysis - the ten types of analysis which can be done (not yet posted).
There are ten types of economic analysis which can be done in regard to organizations, program or other interventions. In outcomes theory these ten types of analysis are grouped on the basis of how much is known about each attribution of changes in high-level outcomes to interventions. This makes it easier to make decisions about what is the most appropriate type of economic analysis depending on the type of information which is available about attribution.
Contracting for outcomes
Despite the calls for, and appeal of the notion of contracting for outcomes, those attempting to do it in practice often find that it a difficult, confusing and frustrating experience. This occurs because people are trying to structure such contracting in ways that do not conform to the underlying principles of outcomes systems outline in outcomes theory. Once such contracting is set up in a way that is consistent with these principles it can be done in a systematic, transparent and robust way.
Attempts are often made to distinguish evaluation from other process. Making such distinctions is difficult to do because such processes (evaluation, monitoring, performance management, assessment, quality assurance) can, on occasion include undertaking some similar activities. A more fruitful way of proceeding is to identify the underlying activities which can be undertaken within these processes and use these to identify and contrast what is being done in a particular situation. A practical way of doing this is discussed.
Reframing program evaluation as part of collecting strategic information for sector decision-making
Program evaluation has tended to have an implicit focus as a stand-alone activity in its own right. However, evaluation can be reframed as simply one part of what should be an integrated tool-kit for collecting strategic information for sector decision-making. Such a reframing puts a different emphasis on the role and conduct of evaluations.
What exactly is best practice? The term, although in wide usage, lacks clarity because it is not clear whether: 1) best practice consists of a claim that practitioners, from their own experience, believe them to be feasible and 'useful' to implement; or 2) that best practice consists of practices which have been proven to improve high-level outcomes (through making a strong outcome/impact evaluation claim). Given the way best practice is used in practice, it may be useful to view best practice as including 1 above and use the term evidence-based practice when referring to 2 above. (It should be noted that it can be argued that the term evidence-based practice should also include more than just 2 above).
The major challenge in the area of best practice is to get it implemented on the ground. The use of visual outcomes models can help practitioners to identify what best practice is; to formally represent best practice; to communicate best practice; and to embed best practice into the planning processes used by practitioners when designing and implementing their programs and interventions.
In most areas of knowledge and practice the attempt is being made to make things easier to understand and to make applied systems easier to work with. The area of outcomes, monitoring and evaluation is no exception. However, in outcomes, monitoring and evaluation there is currently a combination of over-simplification and over-complication which makes working in the area difficult. Progress can be made on this by improving the sophistication of what is being done in those areas where there is currently over-simplification and doing what can be done to make those areas which are highly complex more easy to work with.
The error of limiting focus to only the attributable arises in situations where the focus of action is limited to that for which specific providers can be held to account and this accountability is based on that which is demonstrably attributable to specific providers. This error is often a strategic mistake because it limits the focus of possible action and discourages collaboration. It can be avoided by building outcomes and performance management systems based on the five building-blocks models. This approach provides a framework for providers to show that they are focusing on the links upwards to higher-level outcomes regardless of whether or not changes in those outcomes are demonstrably attributable to them. Accountability is then set for both demonstrably attributable indicators (which tend to be lower-level e.g. outputs) and also for focusing on measuring higher-level indicators and understanding the links between (managing for outcomes) them by using visualized outcomes models (results maps) to structure strategic performance discussions
Single list of indicators problems
A common approach within results-based, performance management and monitoring systems is to identify a single list of indicators and to then attempt to hold a program, organization or other intervention to account for achieving particular levels on these indicators. Where such an approach is combined with an insistence that such a list represent 'outcomes' or 'results' of some sort in contrast to 'just outputs' this approach is flawed. The non-output attributable intermediate outcome paradox means that as you move upwards towards high-level outcomes in compiling such an indicator list, the less likely it is that you will be able to locate indicators which are at the same time non-output plus also being demonstrably attributable to a particular intervention. The solution to this problem is to use an approach which includes two 'lists' or types of indicators - demonstrably attributable ones and non-demonstrably attributable ones.
A paradox arises when non-output demonstrably attributable indicators (technically measurements of what are often called intermediate outcomes) are sought in regard to a program, organization or intervention. In many cases, such things cannot be found. This paradox arises from the demand that intermediate outcomes are found which are: able to be used for accountability and are therefore demonstrably attributable to a particular program, organization or intervention; but at the same time, there is an insistence that such indicators must not also be outputs. The paradox is explained and the way it can be avoidable is discussed.
Implications of an exclusive focus on impact evaluation in 'what works' evidence-based practice systems
Impact evaluation is being widely encouraged in many sectors and countries as a way of providing information for 'what works' evidence-based practice systems. While this is generally a good idea, nuanced 'what works' systems need to take into account the fact that in some cases the types of programs can vary in the ease with which impact evaluation can be undertaken on them. Where this occurs within a 'what works' system a inadequately nuanced system can create a bias in favor of programs for which it is easy to undertake impact evaluation.
'What works' evidence-based practice systems are encouraging the use of impact evaluation to demonstrate attribution of changes in high-level outcomes to interventions. Often neglected in discussions regarding such systems is the potential trade-off between demonstrating attribution and encouraging collaboration. If this trade-off is not managed a 'what works' system can unintentionally create a bias towards atomistic rather than collaborative programs even where collaborative programs may have a greater chance of success.
Tools and checklists
The following section lists tools and checklists which have been developed within outcomes theory and which can be used when doing outcomes, evaluation and related work.
Once it has been decided that using an impact/outcome evaluation design is likely to be appropriate, feasible and affordable in regard to a particular intervention, the next question is working out which is the most appropriate impact/outcome design to use. Making this selection can be complex, particularly for those not familiar with all of the possible options. This article provides a decision-making table approach to selecting one of seven possible groups of impact/outcome evaluation designs. It also provides a set of checklists for each of the seven possible types of impact/outcome evaluation. In addition, it helps in selecting the best technique for attempting to deal with non-equivalent comparison group issues in the pragmatic, and hence frequently used, constructed matched comparison group designs.
Outcomes systems - checklist for analysis
Outcomes systems can be analyzed using the framework provided within outcomes theory. In particular, three aspects of outcomes theory can be used for this analysis: the outcomes system building-blocks model; key features of steps and outcomes; and conventions for visualizing outcomes models. Based on outcomes theory, a checklist has been developed which allows any outcomes system to be analyzed for the common problems that often arise in such systems. Outcomes systems which conform to the criteria in the checklist will avoid the problems faced by outcomes systems which do not. The checklist helps identify the areas in which current outcomes systems which are having problems need to be improved and shows what needs to be done to improve them.
Action research is a particular type of social science research which uses repeated cycles of research/action/research to make sure that action is informed by research. At the same time it seeks to ensure that research does not lose touch with the purpose for which it is being done (i.e. to improve activity on the ground). It has some similarities with other types of 'embedded' research such as formative evaluation (evaluation with the purpose of optimizing the implementation of a program, policy or other intervention). An outcomes modeling approach can be used as the basic tool within an action research project. This has a number of advantages: outcomes models are a good way of deciding on priority action; they can be used to channel the results of previous research into reflections on current action; they can be used as the basis for new research as part of the action research cycle; they can be used to specify what indicators and outcomes should be monitored as part of the action research activity; and they can be used as the basis for those doing the action research to share best practice amongst themselves.
Analyzes of particular outcomes systems
The following is the first in a series of analyzes of particular outcomes systems. An outcomes system is any system which attempts to deal with specifying, measuring, attributing and holding players to account for changes in outcomes of any type. Such system go by a variety of names, such as: results management, performance management, performance measurement, evaluation, evidence-based practice, investment strategies, value-for-money exercises, benchmarking, contracting, strategy etc. In each case they are analyzed using the generic concepts and common language provided by outcomes theory.
Analysis of the Balanced Scorecard/Strategy Map approach shows that it has advantages over more simplistic monitoring systems because it encourages the identification of indicators from a number of domains and it encourages the use of a type of visualized outcomes model. Technical improvements which could be made to the system, based on insights from outcomes theory are suggested.
The United Nations Results-Based Management System is analyzed using the Outcomes Systems Checklist. This identifies a number of technical flaws in the system and shows how they could be corrected by changing the system to make it consistent with the principles of a well structured outcomes system.
Conference papers related to outcomes theory
The following are conference papers related to aspects of outcomes theory.
Encouraging better evaluation design and use through a standardized approach to evaluation planning and implementation - Easy Outcomes (2008 European Evaluation Society Biennial Conference)
Communicating a program evaluation plan to high-level stakeholders is difficult because of the complexity of such plans and the limited time stakeholders have to consider them. However it is important that, in spite of this, high-level stakeholders have the opportunity to consider what evaluation is possible and what evaluation they can afford to fund. A visual evaluation plan presented in a standardized format and visualized in specifically designed software provides a new and highly effective way of assisting high-level stakeholders to make decisions about evaluation priorities. The Easy Outcomes approach provides a standardized format for such plans. In addition, such visual evaluation plans can be used not only to make planning decisions about the evaluation but also to control and monitor its implementation. Further, the outcomes models within such evaluation plans can be used for strategic planning, monitoring, economic evaluation, outcomes-focused contracting and other purposes.
What added value can evaluators bring to governance, development and progress through policy-making? The role of large visualized outcomes models in policy making (2008 European Evaluation Society Biennial Conference)
Visual outcomes models are a useful way of structuring high-level policy discussions. This paper presents outlines the way they can be used by evaluators to add value to the policy making process. For such models to be useful they need to be build according to a specific set of guidelines which make them 'world-centric' rather than the more 'program-centric' models which are often used in evaluating or examining a single program. Once build such models can be used for strategic prioritization, monitoring, evaluation, evidence-based practice, outcomes-focused contracting and other organizational and sector-wide functions.Proving that preventive public health works - using a visual results planning approach to communicate the benefits of investing in preventive public health (Key note address to Keeping Well Population Health Provider Conference 2009)
Preventive public health is now working in an era in which it is increasingly important to be able to clearly put the case for spending public money on prevention. The general argument in support of adequate preventive public health funding is strong - it is a case of 'spending to save'. It is smarter than letting people get sick and spending taxpayer dollars trying to cure them after they have become ill. The desire of any government for a highly innovative and productive workforce will not be met if a significant proportion of the young workforce become obese, unfit and hence less productive. What preventive public health has to do, however, is to clearly explain what it is doing and the evidence for it. A clear framework is needed for discussing action planning, results and evidence with busy stakeholders. Visual results plans can help with this in coordinating activity, setting priorities, collating evidence, structuring contracting, and in monitoring and evaluation.
Please comment on this page
Please comment on this page or at the bottom of any of the pages of any of the articles summarized in this list so that they can be improved through critical examination and reflection.
Citing this article
Duignan, P. (2008-2009). Outcomes theory: List of outcomes theory articles. Outcomes Theory Knowledge Base Article No. 218. (http://knol.google.com/k/paul-duignan-phd/outcomes-theory-a-list-of-outcomes/2m7zd68aaz774/25).
[If you are reading this in a PDF or printed copy, the web page version may have been updated].
Please comment on this page or at the bottom of any of the pages of any of the articles summarized in this list so that they can be improved through critical examination and reflection.
Citing this article
Duignan, P. (2008-2009). Outcomes theory: List of outcomes theory articles. Outcomes Theory Knowledge Base Article No. 218. (http://knol.google.com/k/paul-duignan-phd/outcomes-theory-a-list-of-outcomes/2m7zd68aaz774/25).
[If you are reading this in a PDF or printed copy, the web page version may have been updated].
[Outcomes Theory Knowledge Base Article # 248]
Patrick Cline
Invite as author
Scholarly articles?
Thank you for your interest.
You can find references to conference presentations on outcomes theory here - http://knol.google.c
My doctorate which initiated my work on thinking about having a standardized approach to undertaking evaluation is Duignan, P. Evaluating health promotion: The strategic evaluation approach. University of Waikato, Hamilton, New Zealand 1997. Out of this I developed the Strategic Evaluation approach (http://www.strategi
Having developed this software (http://www.doview.c
My first priority has been to develop an approach, then test it on practical examples (I have tested aspects of outcomes theory in many sectors and setting in the course of my consulting work - which has been occupying most of my time till recently). Then I wanted to get the material up on the internet and present it at conferences so that others could also use it. Now that I have got it to this stage I can turn my mind to getting some academic publications out which summarize aspects of it. Due to the nature of the material and that outcomes theory will develop further, the evolving open access internet knowledge base approach of this Outcomes Theory Knowledge Base is the best way to maintain the core of the material in an up-to-date and accessible fashion. Putting it oup on Knol (a Google system which presumably be around for a long time) means that it should continue to be available.
Thanks for your interest, if you have any time available, post any comments you have regarding aspects of the knowledge base on the relevant pages.
Regards,
Paul Duignan, PhD
EditSaveCancelDeleteDeleteBlock this userReport abusive commentHide report window
Village Idiot
Invite as author
There are ten possible types of economic analysis
Really? Not nine? Not eleven? What a nice round number.
EditSaveCancelDeleteDeleteBlock this userReport abusive commentHide report window