United Nations Results-Based Management System - An analysis

1. The higher the level at which 'results' are specified, the more such 'results' are influenced by external factors and hence the less attributable they are to particular programs and the less sense it makes to hold specific programs (U.N. Divisions in this case) to account for such 'results'. 
2. Difficulties in measuring some 'results'.
3. The system focuses attention just on 'results' related to individual Divisions, not on cross-Divisional and organizational-wide 'results'. 
4. An overly rigid and inflexible approach in which, once performance measures are set (at the start of a program), it is hard to change them to reflect changes in a program due to new circumstances. 
5. The attainment or non-attainment of results is of no discernable consequences to subsequent resource allocation.
6. The system does not work to promote debates about strategic direction.  
7. Many of the expected accomplishments are specified at the level of outputs and the system is largely still based on outputs. 
8. A lack of a sufficiently close relationship between outcomes and indicators. 
9. No prioritization regarding which indicators are more important.
10. Many indicators even though identified, are not being measured. 
11. Multiple overlapping reporting and 'results' systems which creates inefficienciy and frustration and a lack of integration of financial, programtic and staff planning, which are currently separate processes. 

The U.N. system as described in the 2008 report seems to be a system build around a single indicator list. This type of system does not distinguish between two types of result indicators - non-demonstrably attributable ones (which tend to be higher-level and cannot be used to hold a program to account because they are not demonstrably attributable to the specific program) and demonstrably attributable ones which can appropriately be used to hold a program to account. A single indicator list system suffers from the single indicator list problem when it is demanded that  that results indicators be both demonstrably attributable, but also results-focused (i.e. at a high level) and, therefore, by implication, not at the outputs level. As noted in the 2008 U.N. report, the basic tension is that the higher up a results indicator is towards final results, the less likely it is to be possible to attribute changes in it to a particular organizational unit and therefore the less appropriate it is to use it to hold that organizational unit to account. This leads to the search for what can be termed non-output attributable intermediate outcomes and the fact that it may not be possible to find these in many instances is known as the non-output attributable intermediate outcome paradox. 

There are four possible consequences for an outcomes system which is suffering from the single indicator list problem which are described in detail in the article referred to above. In summary these are: 1) giving up and allowing the single results indicator list to become an incoherent mix of attributable and non-attributable indicators; 2) expanding each indicator to include within it both a higher-level 'result' and a lower-level attributable 'output'; 3) giving up on the demand that results indicators have to be attributable, which then allows them to be higher-level, but makes them no longer useful for holding programs to account; and, 4) giving up on the demand that indicators be higher-level and results-focused and allowing them to sink down to lower levels which means that they will be attributable and that they can be used to hold the program to account but that they will disappoint those with the expectation they they be struck at a high-level.  

The U.N. system is suffering from the fourth consequence, the results indicators (referred to as 'expected accomplishments' within the system) are tending to be struck at the outputs level (as discussed in Item 7 below). The only coherent way of dealing with the non-outcome attributable intermediate outcome paradox is to have a 'two list indicator system, this allows for two types of indicators to exist within the system. The first are not-necessarily demonstrable indicators. These are ones which attempt to measure outcomes of relevance to a program, but which, by their mere measurement do not establish that changes in them can be attributable to a program. It is important to measure these outcomes for strategic purposes - they are the things that it is hoped the program will improve and it is important to know whether they are getting better or worse. Then there is the set of demonstrably attributable indicators, those which are able to be attributable to a particular program (U.N. Division). The program can then be held to account for these demonstrably attributable indicators. It may be the case that the program is also asked to measure some non-demonstrably attributable indicators and report on how these are tracking, however it needs to be clear that the program will not be held to account for these. Identifying and discussing these two types of indicators is much easier to have against a visual outcomes model rather than just working directly just with sets of indicators.  How to have this discussion in practice against a visual outcomes model is set out in the article on Contracting for outcomes. 

This is a major problem in systems which have no way of dealing with non-measurable steps and outcomes, as appears to be the case within the current U.N. system. The reality of real-world programs is that is will be easy to feasibly and affordably measure some outcomes; it will be harder to do this with others; and for some it is likely to be impossible to feasibly and affordably measure them. It is important that decision makers are kept aware of which outcomes are, and are not, being measured in regard to a particular program. A system based only on an indicator list cannot easily provide decision makers with this information, it tends to lull decison makers into believing that all of the important things are being measured - particularly in systems which are claiming to be results-focused. This means that decision makers are not clear about the risk they are facing in their decision making which stems from uncertain measurement.

The best way of dealing with this issue is to use an outcomes model as the basis for an outcomes system. When such a model is used, and if it is drawn in a way that allows the inclusion of non-measurable steps and outcomes (i.e. by drawing it according to the Outcomes model standards). It then becomes obvious when indicators are mapped onto the outcomes model as to which steps and outcomes are currently being measured and which are not. If such a model, with indicators mapped back onto the underlying visual outcomes model, is used throughout planning, prioritization and reporting, it means that decision makers constantly have in front of them information about which steps and outcomes are currently being measured and which are not. They can then factor this into their decision making and management of the risk they face from uncertainty.

This problem stems from the way in which the U.N. system is set up. The inherent pressure within the system is for results indicators (expected accomplishments) to be accountable to a particular Division. This then leads to the expectation that they be demonstrably attributable to that specific Division. This again, in turn, leads to pressure for them to be things which are controllable by the specific Division (if something is controllable by a Division it is clearly attributable to the Division so it can therefore be used to hold the Division to account). Something which is controllable by a particular Division is, by definition, not going to be something which a number of Divisions contribute to.

The solution to this problem is to base the results-management system around a visualized outcomes model drawn according to the outcomes model standards. Using these standards produces outcomes models which are focused on modeling the external world on which an intervention is focused. Such models include the highest-level outcomes and can also include within them all of the factors (in this case, both those influenced by U.N. Divisions, and external factors) which affect them occurring. Such models are described as 'world-centric' models in contrast to 'program-centric' models which only model steps and outcomes which are measurable and attributable to a particular program. Such outcomes models can cover the high-level outcomes of a number of different programs. Once such a 'world-centric' model has been developed, individual programs can be mapped back onto it to show the specific steps and outcomes it is believed they will influence. Following this, measures of these steps and outcomes can be identified (these are known as not-necessarily demonstrably attributable indicators) and finally demonstrably attributable indicators can be developed - these are what the individual U.N. Divisions will be held to account for.

This is a common problem in situations where circumstances may change rapidly and programs need to be able to respond just as quickly. An outcomes system needs to be flexible enough so as to be able to facilitate a program changing in this way. Building an outcomes system around a visual outcomes model helps to promote such flexibility in two ways. 

First, if a visual outcomes model is being used and good practice is being followed, the outcomes model will be kept up-to-date. At any point in time, the model should represent the most accurate possible picture of the high-level outcomes which are being sought and of the steps which are being taken to achieve them. Using a regularly up-dated outcomes model in this way provides a basis for talking about the program independently of the set of result indicators being used for the program (known as 'expected accomplishments' within the U.N. system). It is much easier to conceptualize what is happening in a program and how it is changing by using a well-structured visualized outcomes model than by trying to work off an unstructured results indicator list. That is, one which is not set out in any kind of causal hierarchy, such as the list of 'expected accomplishments' which seems to be the basis of the U.N. results-management system.

Working with a visualized outcomes model in this way, stakeholders can agree when the program has changed and this provides a basis for them agreeing that it is reasonable to change results indicators if it has changed. If there is no visual outcomes model against which to test suggested changes in results indicators, then when program staff suggest that there should be a change in one or more indicator, and when they present a revised list of indicators, there is no easy way of assessing whether or not to accept such proposed changes. Do they reflect an actual change in the program (which is easily examined and communicated by using a visual outcomes model)? Or is it just a case of the program staff trying to change one or more results indicators to avoid the program being held to account for not doing things that it should have done. The use of a visual outcomes model makes it much easier to assess the difference between these two instances.

Second, initially developing an outcomes model prior to identifying results indicators for accountability purposes, allows careful consideration of the level of generality at which the steps within the outcomes model should be struck. In situations which are likely to change rapidly, steps within an outcomes model may have to be set at a higher level of generality because the specific details of what will be done cannot be specified in advance prior to the program being implemented. This involves decision makers accepting a trade-off. This trade-off involves them accepting that they will not be able to specify in such detail results indicators which can be used for accountability purposes and will function throughout the life-time of the project. If they want such results indicators which they can use for accountability purposes they will have to revisit them in the course of the program developing. The reality of having to accept this trade-off is immediately and directly communicated to stakeholders when working with a visual outcomes model and they can identify those situations in which it is going to need to be made. In contrast, rushing to develop sets of results indicators for accountability purposes (as in the U.N. system) without the step of building a visual outcomes model, will immediately push thinking downward from the more general and flexible to the more specific and lock in place results indicators which will become out of date as the program has to change in response to changing circumstances.  

The U.N. system is functioning in a complex environment and to some extent this finding of the 2008 report reflects the third type of problem the U.N. system faces (i.e. more general problems specific to the nature of the U.N. and the context in which it works). However, a system which has the technical flaws which the U.N. system appears to have, can result in a loss of stakeholder confidence in the integrity of the system and this can result in it not being used in practice to hold parties to account.

This is to be expected as a natural consequence of the fact that the U.N. system as it is currently set up seems to use a single list indicator approach and therefore suffers from the non-output attributable intermediate outcome paradox.  In this particular case, the consequence has been to drive the results indicators (expected accomplishments) down to the output level as the attempt is made to keep them suitable for holding U.N. Divisions to account.

It is likely that this situation will continue until the system is modified. The best way out of the dilemna is to have a three stage process where: 1) an outcomes model is developed without worrying about measurent or demonstrated attribution; 2) indicators are mapped back onto the model without worrying about whether the are demonstratably attributable; 3) demonstrably attributable indicators are then identified from the full set of indicators; and 4) decisions are then made about which indicators parties will be held accountable for (ususally demonstrably attributable indicators) and which results indicators they will merely measure (without  being accountable for them) as part of strategic level monitoring. 

Once it is accepted that merely insisting that results indicators should be both attributable and high-level in the face of the non-output attributable intermediate outcome paradox will not magically make it possible to identify such indicators, then the way is cleared for a more realistic approach which in various ways puts the focus on high-level outcomes but which avoids the paradox and still provides clear accountability. Using such an approach, large gains can be made by setting it up in such a way that it encourages: 1) a careful consideration and documentation of the outcomes it is hoped will be achieved and the steps which it is believed are needed in order to achieve these; 2) making sure that this is communicated to all staff working on projects; 3) a consideration of evidence and justfication for the links between the steps and outcomes in such an outcomes model; 4) the use of such outcomes models for strategic planning and discussions about resource allocation; 5) awareness of which steps and outcomes in the model are, and are not, measurable; 6) careful reflection on when high-level outcomes evaluation is going to be undertaken; 7) prioritization of evaluation questions; 8) documentation of which evaluation questions are, and are not, going to be answered and why; and, 9) reporting back monitoring and evaluation results against the outcomes model. All of this helps to put the focus where is should belong, on high-level outcomes, but does so in a realistic manner which avoids the current impasse that the U.N. system seems to have found itself in.

This is result of there appearing to be no mechanism within the system for developing the underlying outcomes model and then mapping indicators back onto it. As a consequence, there is no discipline for making sure that indicators do measure high-level outcomes and for identifying those cases where there are high-level outcomes which are not currently being measured. Instances in which there is a lack of measurement of high-level outcomess can be remedied by either introducing an indicator project to attempt to measure the high-level outcome, or just by living with the fact that it is not currently appropriate, feasible and/or affordale to measure it. In this latter case, it is important that the fact that a high-level outcome is not being measured is constantly brought to th attention of decision makers who are doing strategic planning and resource allocation regarding the project. This can best be done by always working off the visualized outcomes model onto which indicators have been mapped whenever doing strateic planning, prioritization and reporting.

This is inherent in systems such as the U.N. systems because of a lack of a practical method for determining which indicators are more important than others. Mapping indicators back onto an underlying outcomes model is a good way of being able to determine which indicators are more important and which are less so. Such an approach immediately shows the relative importance of different indicators because it shows at which level of the hierarchy of steps and outcomes a particular indicator lies.

This may simply be because of lack of discipine or resorces within the system. However, given the problems with the system outlined above, particularly the fact that many results indicators are at a low level, perhaps it reflects a stakeholder feeling that just measuring more of such lower level indicators (if other similar ones are already being measured) is not particularly useful.