Single list of indicators problem

A problem arises in monitoring, results management and performance management systems (known by the generic name of outcomes systems) where a 'single list' approach is used to develop a list of indicators. Outcomes systems are any monitoring, results management, performance management, quality assurance or other system which attempts to identify outcomes, measure them and attribute and hold parties to account for them. The 'single list' approach can cause problems within such systems when two potentially conflicting demands are made of the indicator list. The first demand is that the list of indicators will be used to hold a program, organization or other intervention 'to account' - i.e. it will be punished or rewarded depending on the levels achieved on the indicators. The second demand, is that such a single list of indicators either include, or be limited to, indicators which are: 'above outputs level'. 

The problem with the single list of indicators approach is due to what is called the Non-Output Attributable Intermediate Outcome Paradox (also known as the Non-Output Demonstrably Attributable Indicator Paradox. This paradox means that the attempt to compile a single list of accountability indicators will often fail at the point that the list rises above the output level. The paradox means that often indicators cannot be found which are: 1) demonstrably attributable to a particular program, organization or other intervention; and which are, 2) above the output level. 

The consequence of the operation of this paradox in regard to a single list of indicators approach is that one of  four things can happen:

1. The confusion resulting from attempting to find non-output demonstrably attributable indicators to put in the single indicator list results in incoherence in the system. There is no longer any clarity about whether or not the indicators in the list should all be accountable or not, or whether they should be above the output level.
   
2. The indicators in the list are amended to become 'compound' rather than 'singular' so that they can include both a high-level outcome element and an attributable, hence accountable lower level element (e.g. an output). A compound step, outcome or indicator is one which contains both an end and a means rather than a singular step, outcome or indicator (which is limited to just a single element). In practice, this involves including an output in the indicator to accompany the higher-level outcome. An example of such a compund indicator is, improving health by producing a pamphlet. The technical problem with this is firstly that the indicator then has 'embedded' within it a piece of causal logic. The particular means in the compound indicator then inevitably becomes the only way in which it is imagined the end in the logic can be achieved (i.e. in the case of the example above, the use of a pamphlet to improve health). In addition, there is a tendency to not allow the same means (i.e. an output) to contribute to more than one higher-level outcome. This is an instance of siloing which is where a lower level step in an outcomes model is only allowed to be linked to a single higher-level outcome. This can result in a distortion of the representation of the causal flow in the real world which one should always be attempting to accurately represent within an outcomes system. See the article on Causal models - how to structure, represent and communicate them.
3. Relaxing the constraint that the indicator should be demonstrably attributable. This certainly leads to a list of intermediate outcomes which are at a higher level than outputs. However, because they are not demonstrably attributable to the program, organization or other intervention which is being held to account for them, this can lead to one of three problems. The first is that where such an indicator is not met, a provider can claim that other factors influenced its lack of achievement and therefore attempt to escape from being held to account for it. The second is that a provider can be unfairly punished when such an indicator is not met, in spite of the fact that it has not been met because of other factors. The third is that a provider can be unfairly rewarded when such an indicator is met, but it is not due to anything in particular that it has done since the result has been influenced by other factors. Over time, outcomes systems which have this problem suffer from an undermining of their credibility. 
   
4. Relaxing the constrain that the indicator should be a non-output. This leads to the credibility of the system as an outcomes-focused system being undermined as soon as it becomes apparent to stakeholders that the accountability indicators which are being used are just set just at the outputs level. this
   This is a common source of complaint about contracting for outcomes systems when the attempt is made to implement them.
   

The solution to the single list of indicators problem is to have either two sets of indicators, or (what amounts to the same thing) to have a single list of indicators which are divided into two separate types of indicators.

The two types of indicators which need to be distinguished are:

1. Non-demonstrably attributable indicators - ones which, by merely measuring them, do not establish that they have been caused by a particular program, organization or other intervention.

2. Demonstrably attributable indicators - ones which, by merely measuring them, do establish that they have been caused by a particular program, organization or other intervention.

Once these two types of indicators have been differentiated, then they can then be used for different purposes within the overall outcomes system. The non-demonstrably attributable indicators can be used for strategic purposes to see whether outcomes are, or are not, improving. The demonstrably attributable indicators can be used to hold the program, organization or other intervention to account.  In outcomes systems which use a dual indicator list approach, non-demonstrably attributable indicators are give different names in order to differentiate them from demonstrably attributable indicators (which may in some cases be called performance measures). These name can include names such as: state indicators, environmental indicators, strategic indicators, sector-wide indicators.

There are major advantages when working with such dual indicator lists in mapping them back onto an underlying visual outcomes model. This approach can be used to undertake contracting for outcomes. See the article on Contracting for outcomes for how to do this. 

A single list of indicators approach when used within performance management, monitoring and results management systems can have major problems in an era where there are increasing demands for  accoutability for higher-level outcomes. These problems can lead to an undermining of the credibility of systems which jsut use a single list of indicators approach. The solution to this problem is to differentiate between two types of indicators, those which are demonstrably attributable to a program, organization or other intervention and those which are not. How to do solve the single indicator list problem by using an approach based on a visual outcomes model is set out in the article on Contracting for outcomes.

Please comment on this article

This article is based on the developing area of outcomes theory which is still in a relatively early stage of development. Please critique any of the arguments laid out in this article so that they can be improved through critical examination and reflection.

Citing this article

Duignan, P. (2009).Single list of indicators problem. Outcomes Theory Knowledge Base Article No. 243. (http://knol.google.com/k/paul-duignan-phd/single-list-of-indicators-problem/2m7zd68aaz774/85).

[If you are reading this in a PDF or printed copy, the web page version may have been updated].