Methods and analysis techniques for information collection

Introduction

Methods and Types of Analysis 

 

Evaluation methods should be selected on the basis of whether or not they will be able to effectively answer the evaluation questions within each evaluation project. There are a number of consideration when selecting evaluation methods. The first is to make sure that the methods which are selected are appropriate and ethical for the groups involved in them. Appropriate ethical approval processes need to be followed where required and evaluators need to conform to appropriate ethical standards. Methods selection in evaluation also needs to be very focused on value for money in terms of the selection of methods. There are usually limited evaluation resources and they need to be used wisely. Evaluation methods should be ‘fit for purpose’ in the sense that they provide the amount of information required at a level of certainty which is sufficient for the purpose of the evaluation. There is little point in using all of one’s evaluation resource for a particular evaluation on the perfect application of a single method when it would have been better to use several different methods at a level which was ‘fit for purpose’ rather than attempting to achieve a perfect application of the method. Of course, methods have to be used in a way that will produce usable information. Selecting the mix of evaluation methods for a particular evaluation is therefore a matter of making decisions about the various trade-offs between spending a lot of resources on a few methods, or of undertaking a wider mix of methods.

 

Stakeholder consultation – a wide range of consultation methods are used at different stages within evaluation. In all of the purposes for evaluation (formative, process and impact/outcome evaluation) there is likely to be some type of stakeholder consultation about the evaluation. Stakeholders can be divided into two types, first participant stakeholders – ones who are going to be involved in some way in the process of the evaluation itself (e.g. being interviewed, handing out questionnaires). Second, audience stakeholders – ones who are interested in the results of the evaluation. Any particular stakeholder can be either of these types of stakeholders or both types. In formative evaluation stakeholder consultation may also involve aspects of the program itself in addition to just the evaluation. This consultation may focus on ideas for the design of the program; getting stakeholder support for the program; working out protocols with stakeholders for aspects of the program. Analysis: Qualitative summary of what the workshop concluded

 

Needs assessment – this can involve a wide range of processes for working out if there is a need for the program. It may involve talking with stakeholders, analysing existing data, doing surveys to identify need, looking what other programs and activities are already occurring. The purpose is to define if there is a need and to detail as much as possible what that need is so that the program can be designed to meet the need. Analysis: Quantitative or qualitative analysis.

 

Outcomes logic model drawing – drawing outcomes models of any type that are then used to clarify thinking about all of the steps which are needed in order to achieve higher-level outcomes. These  models are known by different names including: logic models, program logics, intervention logics, strategy maps, means-ends diagrams, results chains etc. There are often different conventions about what types of steps and outcomes are allowed in such models (e.g. do they need to be measurable and attributable). A set of guidelines for drawing such models which can be used within the Easy Outcomes approach is provided earlier in this workbook. Once drawn, the links between steps and outcomes within an outcomes model can be examined to see if there is evidence for the link.  Analysis: Logic analysis of whether the pattern of causality claimed in the outcomes model makes logical sense.

 

Collaborative group processes - Often used in formative evaluation (evaluation with the purpose of improving the implementation of a program). For instance, if a program is being run in 20 localities across a country the program workers could be brought together either physically and/or electronically to work in an ongoing collaborative style with formative evaluators and in some instance with the funder. This collaborative process would: define common and locality-based outcomes; identify indicators for tracking success; working in groups to define strategies; develop collective resources; collectively approach national level stakeholders to remove policy barriers; provide collegial support for each other; and document the process for evaluation purposes (process evaluation – describing the course and context of a program).  Analysis: Qualitative summary of what the workshop concluded.

 

Review of: literature, previous experience – A review of the literature is a summary of the research findings from the past which relate to the program. Literature reviews can range from a fairly informal look at what research has been published about the area a program is working in, to highly structured reviews which are called Systematic Review. In areas in which there is quantitative data available these systematic reviews attempt to summarize all of the outcome evaluation findings from the past (examples of systematic reviews can be found at http://www.cochrane.org for the health sector and http://www.campbellcollaboration.org for the social sector and http://www.cebc.bham.ac.uk for the natural resource conservation area). In addition to research literature reviews, there can be reviews of previous experience with programs similar to the program which is being evaluated which has not got into the research literature. This is often done by an evaluator (in the case of formative evaluation) contacting other programs to find out what they have learnt about best practice for running such programs. In the case of a number of programs which are running concurrently some sort of Collaborative group process as described above can be used. Analysis: Qualitative summary of the research literature or previous experience. Quantitative analysis of the effect size from previous studies in the case of some systematic reviews.

 

Records collected not just for the evaluation – In any program there are likely to be administrative or other records which are routinely collected. These would be collected regardless of whether the evaluation is taking place or not. These are often in the form of statistics about those who are involved in the program or written notes about those involved. Analysis: These records can be analysed using qualitative or quantitative analyses.

 

Photo, video or audio recordings of program activities – As part of the evaluation various types of recordings of program activities may be made using photo, video or audio recording. This material may then be analysed using qualitative or quantitative analyses as part of the evaluation. In addition to whatever technical analysis is undertaken of these types of recordings, visual or audio recordings are a good way to quickly communicate to an audience what a program consisted of and they are often used in explaining the program to participants at the time that evaluation results are being disseminated. Analysis: Qualitative or quantitative analysis of the material produced.

 

Other records of the program made for the evaluation – In addition to the two types of records discussed so far, there are other records which are made specifically for the evaluation itself. This covers a whole range of records which are then analysed qualitatively or quantitatively as part of the evaluation. Analysis:

 

Document analysis – analysis of documents in order to extract themes in regard to the topic being evaluated. This can include all types of documents such as formal reports, minutes of meetings, memos and print media reports or electronic media transcripts. The analysis can range from a extraction of general themes to a tight, specific and detailed analysis. It may use either quantitative analysis (e.g. counting the number of times that a theme occurs) and/or qualitative analysis (e.g. identifying the major themes). In either case it is important to document the criteria which are used to make decisions about rating and analysing material. In addition in some evaluations material is independently rated by more than one rater so as to increase the accuracy of the rating. Analysis: Qualitative or quantitative analysis of the documents being examined.

 

Participant questionnaires/scales/surveys:  written, postal, web-based – Evaluations use a wide range of different types of questionnaires/scales/surveys (sometimes called instruments) which can be delivered in different ways (written, postal, web-based). Many different types of simple instruments are used in evaluations. More complex instruments may require the involvement of those with expertise in their design and use. Sampling and analysis of these instruments may also require considerable expertise. Analysis: Quantitative or qualitative analysis.

 

Participant (P) interviews, P Group interviews: face to face, telephone (audio/video record?) – Participant interviews can be used in formative, process and impact/outcome evaluation. Such interviews can use highly structured questions through to broad ‘semi-structured interview schedules’ which are just lists of question areas to be covered in the interview. In some styles of interviewing the interviewer does not have to follow the exact order of the questions on the interview schedule as long as the participant covers all of the questions over the course of the interview. Participants may be asked a wide range of questions directed at assessing the effect of the program on them. They may also be asked about the program more generally (for process evaluation); to make suggestions for program improvement (for formative evaluation); or they may be asked to make judgements about whether the program was an overall success  for them(for impact/outcome evaluation). If participants are doing the latter it is a good idea for them to be given some sort of success criteria so that all participants can use the same type of criteria to judge the program. Participants and other stakeholders may also be involved in developing the success criteria. Face to face participant interviews usually provide the best information, however telephone interviews can provide adequate information and have the advantage of costing less, being able to be done within a shorter time frame, and usually being more convenient for the participant and the interviewer. There are some circumstances, e.g. because of cultural reasons, that face to face interviews would be preferred over telephone interviews. A set of participant interviews may be repeated in an evaluation, for instance at the start, in the middle and at the end of a program. Telephone interviews generally make the burden on participants less than face to face interviews and therefore may be more likely to have a high response rate. However, this depends on telephone coverage and ease of using the telephone amongst the participant group. Interviews allow interaction between the interviewer and the participant which allows the interviewer to probe participant replies. It also means that the interview can take up at later interviews some themes which have emerged at earlier interviews if this fits with the protocol being used in regard to the participant interviews. Group interviews are where a number of participants are interviewed together in the interests of efficiency. Once one participant has made a point the others can just quickly agree or disagree with it. This is more efficient but has the downside that it may produce a less divergent picture of the program. It may be possible to supplement it with a private questionnaire about the program being filled out at the start of the group interview before the convergent pressure which comes from people being in a group situation starts to operate. Group interviews are different from focus groups discussed below in that the emphasis in a group interview is not in generating a lot of discussion between the participants, just getting their own views from them as efficiently as possible. Analysis: Participant interviews are usually analysed in a straight forward manner with the key themes identified by the group of participants being extracted and summarized. More complex qualitative research analysis methods may be used in some cases.

 

Non-participant observation – non-participant observation covers a wide range of types of observations which can be made to get information about a program. Some of these are referred to as unobtrusive observation and include measures because taking the measurements is unlikely to influence behaviour (e.g. counting the amount of recycling put into recycling bins at an event). Analysis: often quantitative analyses are used in regard to this type of observation.

 

Participant observation – participant observation in this context does not refer to observing participants but rather to a method used in evaluation research where the evaluator becomes a participant in the program they are evaluating. Such involvement has the potential to influence the way that others react depending on how the evaluator operates. However, it has the advantage that the evaluator can sometimes get a much closer picture of exactly what is happening in the evaluation than then may if they did not involve themselves in the program. It is sometimes possible to take this sort of approach at only one or several sites on which a program is operating so that the evaluator gets a better picture of how the program operates but also has data from sites where they were not involved. This type of approach often provides good insights into the more informal, undocumented aspects of what is happening in a program. Analysis: the analysis used for this method tends to be qualitative.

 

Key informant (KI) interviews, KI group interviews: face to face, telephone (audio/video record?) – Key informants are people who are in positions which mean that they are likely to have particular knowledge of a program. They are often interviewed in formative, process and impact/outcome evaluation. In such interviews, key informants can be asked for their judgements about any aspect of the program. Key informant interviews usually work off a ‘semi-structured interview schedule’ which is list of questions about the program. In some styles of interviewing the interviewer does not have to follow the exact order of the questions on the interview schedule as long as the key informant covers all of the questions over the course of the interview. Key informants may be asked to describe the program (for process evaluation); to make suggestions for program improvement (for formative evaluation); or they may be asked to make judgements about whether the program was an overall success (for impact/outcome evaluation). If key informants are doing the latter it is a good idea for them to be given some sort of success criteria so that all key informants can use the same type of criteria to judge the program. Key informants and other stakeholders may also be involved in developing the success criteria. Face to face key informant interviews usually provide the best information, however telephone interviews can provide adequate information and have the advantage of costing less, being able to be done within a shorter time frame, and usually being more convenient for the key informant and the interviewer. There are some circumstances, e.g. because of cultural reasons, that face to face interviews would be preferred over telephone interviews. A set of key informant interviews may be repeated in an evaluation, for instance at the start, in the middle and at the end of a program. Telephone interviews generally make the burden on key informants less than face to face interviews and therefore may be more likely to have a high response rate. Interviews allow interaction between the interviewer and the key informant which allows the interviewer to probe key informant replies. It also means that the interview can take up at later interviews some themes which have emerged at earlier interviews. Group interviews are where a number of key informant are interviewed together in the interests of efficiency. Once one key informant has made a point the others can just quickly agree or disagree with it. This is more efficient but has the downside that it may produce a less divergent picture of the program. It may be possible to supplement it with a private questionnaire about the program being filled out at the start of the group interview before the convergent pressure which comes from people being in a group situation starts to operate. Group interviews are different from focus groups discussed below in that the emphasis in a group interview is not in generating a lot of discussion between the key informants, just getting their own views from them as efficiently as possible. Analysis: Key informant interviews are usually analysed in a straight forward manner with the key themes identified by the group of key informants being extracted and summarized. More complex qualitative research analysis methods may be used in some cases.

 

Key informant questionnaires/scales/surveys: written, postal, web-based – questionnaires/scales/surveys in written, postal or web-base form may be used in evaluations. They are used for the same reasons as discussed for key informant interviews above. In contrast to interviews these methods do not allow for the interviews to probe replies in the same way. Analysis: Analysis may include quantitative analysis of results as well as qualitative analysis of themes.

 

Focus groups: participant, key informant – Focus groups are a method where the main emphasis is on collecting information which arises from the interaction between those involved in the focus group. They are based on the idea that it is in interaction between those in the group that a deeper analysis will evolve of the topic under discussion. They can be used at various stages in an evaluation. In formative evaluation they are often use to pre-test aspects of a program with potential participants. Analysis: The usual method of analysis is a simple qualitative analysis of the themes which emerged from the focus group illustrated with representative quotes from participants. 

 

Stakeholder assessment workshops – Stakeholder assessment workshops are a process where a group of stakeholders are asked to assess the success or otherwise of a program. In order to do this they need to have a clear set of success criteria. The idea is for the group as a whole to come to a consensus on the success of the program. Obviously the use of such a method depends on the politics of the context in which a program is operating and the chances of getting a group of stakeholders which is willing to make such a judgement, and which is capable of making such a judgement in an unbiased way. Analysis: Qualitative summary of what the workshop concluded.

 

Expert assessment (e.g. clinical examination) – There is a wide range of different types of expert assessments which can be made in an evaluation. These could, for instance, be clinical examinations, psychological or educational assessments or any other expert professional assessment. Analysis: analysis can either be quantitative or qualitative.

 

Case studies – Case studies are where there is a particular focus on one or more instances within an evaluation. These can be individuals, particular instances in which a program is being implemented or particular settings. Case studies typically use a more in-depth assessment of all aspects of the instance being studied. They are often used to complement measures which are being used in evaluations across a wider number of instances. Analysis: Qualitative or quantitative analysis can be used.

[Outcomes Theory Article # 219]

There is a range of different types of information collection which can be used in regard to outcomes systems (evaluation, monitoring, strategic planning etc.). In addition this data can be analyzed using either quantitative or qualitative techniques.