Distinguishing evaluation from other processes (e.g. monitoring, performance management, assessment, quality assurance)

A topic article in the Outcomes Theory Knowledge Base

Attempts are often made to distinguish evaluation from other processes (e.g. monitoring, performance management, assessment, quality assurance). Making such distinctions is difficult to do if it does not have available a conceptually tight framework for identifying the activities which are undertaken within these processes. This is because there are a number of activities which are undertaken within more than one of these processes. The most efficient approach to this issue is to simply identify the activities which are being done in a particular situation in such a way that it provides an exhaustive specification of what is being done. The verbal labels which are then used to label these sets of activities will vary depending on the the situation and the way stakeholders wish to use particular terms like evaluation, performance management, assessment and quality assurance. This is a topic article in the Outcomes Theory Knowledge Base.

Introduction

The attempt is often made to draw a distinction between evaluation and other related processes such as: monitoring, performance management, assessment, quality assurance etc. The purpose of such attempts is to specify particular processes so that situations in which they are being undertaken can be accurately described and distinguished. This article argues that relying on controlling a definition of evaluation (or of any of the other process names) is not a particularly useful approach for two reasons.

  1. Many different stakeholders trained in different disciplines use these terms in different ways for disciplinary, political, administrative and historical reasons and this diversity of usage is likely to continue.
  2. More importantly, there are activities which are undertaken within these processes which are common to more than one process. This means that there is considerable overlap between these processes and therefore they are not fully mutually exclusive.

An analogy

Attempting to define and control verbal labels such as evaluation, monitoring, performance management, assessment etc can be compared to a situation in which someone is building a house and keeps insisting that they need to stop and, from a theoretical point of view, define the absolute difference between the kitchen and the dining room. Now, some people do things in the dining room that others do in the kitchen, and some do things in the kitchen that others do in the dining room. Still other people have a kitchen/dining area where they do both kitchen and dining room things. A more fruitful approach in designing such a house would be to put aside for a moment attempting an absolute definitional distinction between a kitchen and a dining room. You would then specify all of the activities which need to be done, identify where they are going to be done and then design rooms accordingly. For the purposes of building the house the rooms could be named Room A and Room B or any other arbitrary names. After the house has been built, people can call the respective rooms whatever they like. 

Using an approach based on identifying the underlying activities

Applying this approach to the case at hand, a more efficient way of working with the issue of attempting to accurately identify what is happening within processes (such as evaluation and performance management) is to identify the set of underlying 'activities' which can be done within such processes. This set of activities can then be used to specify what is happening in a particular situation. This means that one can then be more relaxed about the verbal labels (evaluation, monitoring, assessment etc,) stakeholders will use to describe what is being done in a particular setting. Stakeholder usage of these terms for describing different processes will be determined by their specific disciplinary background, political, administrative and historical factors.

The five building-blocks of outcomes systems - a way of specifying activities within evaluation, monitoring and other processes

Outcomes theory (the study of evaluation, monitoring, performance management, assessment and related systems) identifies a set of five building-blocks which underlie all outcomes systems. These five building-blocks can be used to identify the main activities undertaken in evaluation, monitoring and related processes. The five building-blocks are set out in Figure 1 below.


Figure 1: The Five Building-Blocks of Outcomes Systems

The five building blocks identify five distinct 'activities' which can be undertaken within outcomes systems (evaluation, monitoring, performance management, assessment and related systems)

  • Working out how it's believed a program or intervention is meant to work - this is done by making explicit the outcomes model (logic model, program logic, program theory) of the program. Such models set out all of the lower-level steps which are needed to achieve higher-level outcomes. Such models can be in someone's head, in a narrative document, in a mathematical model, or in a drawn model (or in all of these) in the case of any particular outcomes system. (Building-Block 1).
  • Routinely measuring indicators of progress toward achieving steps and outcomes - for any outcomes model, indicators may or may not be able to be identified which can measure steps and outcomes within a model. Tracking those that can be measured, lets us find out whether we are on course to achieving the steps and outcomes we want achieved within an outcomes model. However, just measuring the progress of such indicators, while important, does not necessarily say anything about why they are improving (or not improving). Therefore such indicators are called not-necessarily demonstrable indicators. This means that their mere measurement is not taken as having demonstrated that changes in them can be attributed to being caused by a particular program. (Building-Block 2).
  • Identifying those routine indicators which can be demonstrated as attributable to a particular program - it is useful, where possible, to be able to attribute change in indicators for steps or outcomes to a particular program or intervention, this may or may not be possible. For some indicators, the mere fact that they change is accepted as evidence that such change can be attributed to a particular program. These are called demonstrably attributable indicators. These tend to be indicators for lower-level steps within an outcomes model. The traditional measurement of outputs (the goods and services produced by a program) are examples of such demonstrably attributable indicators. (See discussion of the features of steps and outcomes within outcomes models for more information on this). (Building-Block 3).
  • Doing analysis or experimentation to attribute change in high-level outcomes to a program- where routinely collected demonstrably attributable indicators (Building-Block 3) are not available for high-level outcomes, the attempt is sometimes made to use analysis or experimentation to answer one or more high-level outcome evaluation questions. High-level outcome evaluation designs are used for this and seven possible such designs are identified within outcomes theory. One or more of these designs may or may not be appropriate, feasible and affordable in the case of a particular program. (More information on these designs here). (Building-Block 4).
  • Describing, improving and implementing outcomes models - another set of evaluation questions, non-outcome evaluation (formative and process evaluation) questions, are used to describe the course and context of a program (process evaluation) and, in the case of formative evaluation, to improve the implementation of the outcomes model in a particular case. (Building-Block 5).

Application of the approach suggested here

The approach suggested here can be applied to solve the problem which the attempt to differentiate evaluation at a conceptual level from other activity is attempting to solve. For instance, a funder may be insisting that a provider 'does evaluation not just monitoring' or 'does evaluation not just performance management', or 'does impact evaluation not performance management'. 

In such a case, you would get the funder to look at the Five Building-Blocks model in Figure 1, ask them which building-blocks they think are already being done and to identify the additional building-blocks they want to provider to undertake.

Equally, the approach suggested in this article can be used to formally define the myriad processes which are used throughout the world to undertake evaluation and monitoring type activity but which sometimes do not even use the terms evaluation and monitoring in their titles. In contrast, the attempt to compare systems by using high-level process terms such as evaluation, performance management, monitoring, assessment is not particularly useful for the reasons set out in this article. 

Conclusion

An alternative to the continuing debate attempting to definitionally distinguish processes such as evaluation, monitoring and performance management and other terms has been offered based on using the five building-blocks of outcomes systems model to identify the underlying 'activities' which can take place under such high-level process headings.


Creative commons copyright

The intellectual property in this article is covered by a creative commons copyright. This means that you can use what is here for any purpose as long as you fully acknowledge the source using the citation below. If you have any questions, please email paul (at) parkerduignan.com.

Please comment on this article

This article is based on the developing area of outcomes theory which is still in a relatively early stage of development. Please critique any of the argument laid out in this article so that they can be improved through critical examination and reflection.

Citing this article

Duignan, P. (2009). Distinguishing evaluation from other processes (e.g. monitoring, performance management, assessment, quality assurance). Outcomes Theory Knowledge Base Article No. 229). (http://knol.google.com/k/paul-duignan-phd/distinguishing-evaluation-from-other/2m7zd68aaz774/57).

[If you are reading this in a PDF or printed copy, the web page version may have been updated].

[Outcomes Theory Article # 229]

References

  1. Note that while building-blocks 4 and 5 use the term 'evaluation' this is just to make the framework quickly understandable by evaluators and those who know the evaluation distinction between impact/outcome evaluation and process and formative evaluation. Building-blocks 4 and 5 could be formally defined without using the term evaluation. For instance, 4 could be defined as: 'using specific studies to establish attribution in cases where demonstrably attributable indicators do not reach to the top of the outcomes model'. Building-Block 5 could be defined as: 'Using specific studies to better elaborate the outcomes model and its context in the case of a particular program and/or to assist in the effective implementation of the lower levels of an outcomes model in the case of a particular program.

Comments

©2009 Google