Guide to Rapid Impact Evaluation

The theory of Phase 1

Three main groups are consulted in RIE:

1. program stakeholders
2. subject matter experts
3. technical advisors

Because a key risk facing an evaluation is program-centric bias, the goal is to provide a wide variety of perspectives on program outcomes and impact. If only sources from within the program are used for the program summary or data collection, there may be views or perspectives that are missed. This is why interests are intended to encompass all perspectives on the program, and why external subject matter experts are also asked to assess the program’s impact.

RIE uses external subject matter experts to help offset this bias and improve the legitimacy and credibility of the evaluation assessments. Constraints on time or contracting may necessitate using experts from inside government but who are still outside the department where the program operates. Note that only program stakeholders and technical advisors are consulted in Phase 2 of the RIE to help strengthen the program summary; all three expert groups are engaged in Phase 3 to assess the program’s incremental impact.

Identifying appropriate subject matter experts is an important part of the process. Who these are and how they can best be identified will depend on what is being evaluated. Generally, academia, other government departments and research institutions may all be good places to begin looking for people who have knowledge of the subject but who have not been involved with the program itself.

An important decision in Phase 1 is whether to contract with some of the experts. The earlier experts can be identified and the contracting process (if necessary) can begin, the more rapid the evaluation can be. The best approach will vary by department, but it may be useful to explore what options are available, for example:

• sole-source, low-dollar value contracts
• professional services and pre-qualified suppliers or other pre-approved pools of expertise
• memoranda of understanding
• casual contracts
• any other applicable contracting mechanisms

A relevant model might be how focus groups are engaged in your department.

Program stakeholders are divided into subgroups called interests. Interests should include all individuals or organizations that can either affect the program or are affected by it. Each subgroup has a different interest in the program, and their priorities are shaped by that interest and their perspectives. For example, program stakeholders might divide into three different interests:

1. program managers
2. program employees
3. program beneficiaries

All three are consulted as part of the program stakeholder expert group, but they may have very different perspectives on the program’s impacts. Depending on the program, other interests might include partners for implementation, families of program beneficiaries, senior executives and others.

There is no one correct way of dividing stakeholders into interests. How they are divided will depend on the program being evaluated and the sample size. Evaluators may choose to discuss their division of interests with program stakeholders and look at other pertinent evaluations or literature. A sample of the interest groups should be consulted when the program summary is being developed, and all interest groups should be consulted during data collection. Doing so will help ensure that the program summary is not program-centric or biased because only program employees were consulted.

One particular area of difficulty for the pilot projects was identifying technical advisors. In general, technical advisors are experts in the broader context of the program or in a relevant technical or scientific aspect. They need not be familiar with the program itself. In effect, technical advisors are similar to subject matter experts, but with the expectation they will engage more systematically with the evaluation to help bring relevant technical knowledge into the evaluation’s design and analysis. Such advisors can help the evaluator develop the program summary in Phase 2, including the direct outcomes, theory of change and the scenario-based counterfactual. Technical advisors also play a role as an expert group to assess impact. Since such advisors are often not familiar with the specific program being evaluated in advance, their involvement in the program summary also helps them develop familiarity with it.

What exactly makes for a suitable technical advisor will depend in part on the program being evaluated and the needs of the evaluation. For example, for an evaluation of a fisheries management project, the technical advisor could be an ecologist or a social scientist such as an organizational specialist. The Public Health Agency of Canada used a scientific expert in its pilot project.

Not all evaluations may need a technical advisor, particularly those for which the evaluator already has expertise in the science or specific nature of the program. That said, a technical advisor can be a useful source of triangulation and can often bring insights that would be otherwise missing from the evaluation.

The theory of Phase 2

The program summary is quite different from the standard program profile developed in evaluations, as it focuses more on how the program is actually delivered in practice. It is both a product and a process.

As a product, the summary provides the basic building blocks for the evaluation, which are:

• the basis to define the impacts
• the interests involved with the intervention
• the theory of change
• the counterfactual(s)

The program summary is a key part of designing the evaluation, and also helps the RIE be “rapid.”

As a process, the development of the summary helps engage key interests in the evaluation. This engagement helps build legitimacy for the evaluation, promotes high-quality responses to the survey(s) and increases the likelihood the evaluation will be used after it is complete. The summary also helps refine the evaluation questions and impacts so that they are relevant to all interests and provide insights into the program’s causal mechanisms. The summary also confirms or identifies the indicators being measured to respond to the evaluation questions.

The evaluator in Phase 2 consults with two of the three groups of experts involved in an RIE:

• program stakeholders
• technical experts

This consultation will ensure that the evaluation and its questions are developed with a complete understanding of the program. The perspectives of subject matter experts are less critical because they are less familiar with the program. Technical advisors are consulted in this phase to help provide context and background on any scientific or specialist knowledge relevant to the evaluation.

Depending on what was decided in Phase 1, engagement with experts can be done through individual or group interviews (in person or by phone) or by sending out the material and requesting their feedback. Interviews need not be long but should seek to confirm that the summary is a reasonable representation of the program or of where changes are needed. Stakeholders may also have suggestions or contributions to make in developing the counterfactual. Where changes are suggested, including unintended outcomes, these should be incorporated into a revised summary that is recirculated and discussed.

Particular emphasis should be given to confirming the validity of the scenario-based counterfactual and ensuring that it represents a plausible, feasible and legal counterfactual. The counterfactual may not be the preferred option for all the key stakeholders, but all should find it acceptable.

Evaluations must also balance the need to consult with the need to proceed with the evaluation. Evaluators should do their best to ensure that everyone consulted agrees with what is developed, particularly the counterfactual. In some cases, it may be necessary to recognize disagreements and continue with the evaluation, highlighting those disagreements in the report as necessary.

A key part of the program summary is the focus on impacts and outcomes. Briefly, direct outcomes are the changes that have directly occurred as a result of a program. A skills-training program might help people demonstrate increased confidence, for example, or help them find jobs. Impacts are more long-term or high-level effects, and incremental impacts are the difference between what happened with the intervention and what would have happened in the counterfactual.

The goal in developing the program summary is to ensure that all possible outcomes, including ones that may not have appeared in the original logic model or in a Treasury Board submission, are addressed. Experts may identify some unintended outcomes that are not present in the logic model, which can help inform later phases of the evaluation. Such identification and exploration of unintended outcomes can be an important benefit of an RIE.

An approach taken in one of the pilot projects was to provide a list of all the immediate and intermediate outcomes in the program’s logic model to stakeholders and ask them if:

• they expected the program to achieve those outcomes
• there were any other outcomes they had observed or expected to observe that were not listed

Providing such a list and asking these questions can make it easier for experts consulted to provide comments. The evaluation can then discuss any outcomes that stakeholders believed had not been achieved and provide a clear list of unintended outcomes, which can then be explored more fully during the evaluation.

Finally, the program summary is also where the theory of change is developed, if the program does not already have one. As with any theory of change, it should help explain how the program’s activities and outputs will lead to the relevant outcomes, and how each level of outcome will influence the subsequent level of outcome. Theories of change can be useful in any evaluation by pinning down the logic that motivates the program and how the program’s activities lead to its outcomes.

A theory of change is particularly important in RIE because experts will be asked to assess the probability and magnitude of the impacts of the program, and their knowledge and understanding of the program logic will play a key role. Ensuring there is consensus on the program logic and the key mechanisms of change during the development of the program summary will help avoid the problem of different experts assuming that there are different mechanisms of change.

An example of the theory of change for NRCan’s evaluation is available online.

If an RIE is conducted as a line of evidence in a larger evaluation, Phase 2 may be a good time to prepare other parts of the evaluation. For example, if the evaluation will also look at alternative delivery methods or improving process or implementation, it may be useful to engage with experts about those subjects as well.

Example from NRCan’s ecoENERGY for Alternative Fuels program

The following example was used by the pilot project at Natural Resources Canada (NRCan) to assess the probability and magnitude of each of the direct outcomes as identified in Phase 1. After respondents had assessed probability and magnitude for each outcome for the actual program, the counterfactual was explained to them, and they answered the same questions with respect to the counterfactual.

Please indicate the extent to which you believe NRCan’s ecoENERGY for Alternative Fuels program has resulted, or will result in the following outcomes in terms of:

• The likelihood that the outcome occurred or will occur, using a scale of 0 to 4, where 0 means has not or will not occur, and 4 means has or definitely will occur;^{Footnote 3} and
• What the size of the outcome is or will be, using a scale of 0 to 4, where 0 means none and 4 means large.

The purpose of this question is to assess the extent to which the program has led or will lead to the outcomes (i.e. attribution of outcomes to the program) and the magnitude (or potential magnitude) of the outcomes. It is essential that you rate both columns.

The theory of Phase 3

Phase 3 of an RIE begins the engagement with external subject matter experts. Using external experts helps offset program-centric bias in the assessment of impacts, and helps improve the legitimacy and credibility of the evaluation assessments. In some cases, it may be sufficient to use experts from inside government, although at minimum the experts should be outside the department where the program operates. In some evaluations, including in one of the pilot projects, the external perspective was very different from the other groups of respondents. Such perspectives can suggest a potentially interesting further line of inquiry or area for research for the evaluation.

Program stakeholders and technical advisors can be engaged through a web survey or interviews to obtain their assessment of impact. Some flexibility may be required, depending on who is being consulted. For example, in consulting with stakeholders, some senior managers may find it easier to respond in an interview rather than a survey due to time constraints.

The RIE method recommends that subject matter experts be engaged through a facilitated workshop, either in person or via tools such as online conferencing. Workshops can be valuable because they provide an opportunity to ensure that external experts understand both the program and the counterfactual, increasing the validity of the evaluation. Workshops can also allow for interaction between subject matter experts, helping clarify areas of concern or confusion. However, workshops may also create an opportunity for some experts to affect the decisions of others. The facilitator should therefore ensure that such influence does not occur, and that each expert provides their own assessment of the probability and magnitude of each impact, informed but not determined by the discussion with others.

All three federal government pilot projects used surveys instead of workshops to engage subject matter experts because of timing and logistical constraints. Using surveys has not been tested in the way that workshops (both online and in person) have, and so may add a further variable to the evaluation. Future evaluations within government should consider what suits the evaluation best. In order to ensure that the consultations are comparable, however, the same questions should be asked of all experts, even if the method varies.

It can also be useful to ask experts to assess how important each outcome is to the overall effect of the program. Understanding how experts assess the relative importance of different outcomes can be useful in interpreting and understanding the evaluation findings. If the evaluation decides to ask for experts’ assessments, these should be asked for in Phase 3 to avoid having to consult with experts again in Phase 4. For example, experts could be asked how important each immediate outcome is to the intermediate outcomes, and how important each intermediate outcome was to the ultimate outcomes. If the evaluation plans to assess overall impact, as discussed in Phase 4, such assessments can help inform that process.

Calculate the mean assessment of probability and magnitude for each expert group

1. Assemble information from the surveys, the facilitated workshop, interviews and any other quantitative data-gathering methods in a spreadsheet and classify responses by the expert group or stakeholder subgroup they represent. Among other information, you should have from each expert an estimate of the probability of the program having a given outcome and the magnitude effect if it occurs.
2. For subject matter experts and technical advisors, find for each group and each outcome the mean estimate of probability and mean estimate of magnitude (see Table 2).
3. For program stakeholders, calculate the mean responses by subgroup or interest. This is done to ensure that the assessment of outcomes of any one subgroup does not outweigh that of other subgroups based on sample size (see Table 2).^{Footnote 5}

You now have a mean of probability and magnitude for each outcome from the following:

• subject matter experts
• technical advisors and
• each subgroup of the program stakeholders

Impact attributable to the program (Table 2)

4. For each group or subgroup, calculate the expected change in each direct outcome under the program. The expected change will always be a value between 0 and 1.
   • To calculate the expected change in a direct outcome, multiply the assessed probability of the outcome by the assessed magnitude of the outcome. The result can then be normalized to between 0 and 1 by dividing it by the maximum score allowed by the scales. On a scale of 0 to 4, the maximum score is 16; on a 10 point scale, the maximum score would be 100.
   • Thus, if for Outcome A subject matter experts estimated the probability of Outcome A occurring to be 3 out of 4 and the magnitude if it did occur to be 2 out of 4, the expected change in Outcome A would be (3 × 2)/16 = 0.375.
5. Repeat the calculations in step 4 to obtain the expected change in each outcome under the scenario-based counterfactual, or under each counterfactual if using more than one.
6. Calculate the incremental impact or marginal change in each outcome by subtracting the expected change in the outcome under the counterfactual from the expected change in the outcome under the program. The result is an estimated percentage change in the outcome that is attributed to the program, as assessed by that expert group: the program’s incremental impact with respect to that outcome.
   • For example, if the expected change in Outcome A from the program was 0.375 and the expected change in Outcome A from the counterfactual was 0.2, the incremental impact of the program on Outcome A is 17.5% (0.375 - 0.2).
   • In Table 2, the assessed incremental impact of the program on the outcome is 3% and 25% for the program beneficiary group and program management group, respectively.
   • It is also useful to calculate the incremental impact on each outcome assessed by each individual respondent, particularly individual subject matter experts, by:
     • multiplying their assessed probability and magnitude
     • dividing by the maximum score allowed
     • taking the difference between the program and counterfactual, as above

     You can then compare the assessments of individual experts to each other and the assessments of other groups, making it easier to identify trends and outliers in the data.

7. Finally, to obtain an assessment from the program stakeholder group as a whole, calculate the incremental impact on each outcome for all subgroups.

At the conclusion of these calculations, you will have assessments of the program’s incremental impact on each direct outcome from each of the three expert groups, in other words, a range of the program’s likely impacts.

Table 2 provides an example of the calculation of two stakeholder subgroups’ (program beneficiaries and program management) assessment of incremental impact on Outcome A. A similar calculation would also be done for all other outcomes and expert groups. Using mean responses allows equal weighting of the program manager subgroup and the beneficiary subgroup, even though more program managers were consulted. Subject matter experts and technical advisors need not be divided into subgroups but otherwise are calculated similarly.

Table 2 - Text version

Table 2 presents a sample calculation of direct Outcome A for the program stakeholder group. It presents information for respondents A through F, including their stakeholder group; and their estimate of the probability of and magnitude of change in Outcome A occurring under the Program and the counterfactual.

It then presents seven steps for calculating the program stakeholder group’s assessment of incremental impact. First, the information is placed in the spreadsheet. Second and third, the mean assessment of probability and magnitude is calculated for each stakeholder subgroup. Fourth, the probability and magnitude are multiplied together and divided by 16 to get the program’s expected change in outcome C. Fifth, Step 4 is repeated for the counterfactual. Sixth, the impact is calculated by finding the difference between the program’s expected change in outcome and the counterfactual’s. Finally, the entire stakeholder group’s assessment is calculated by averaging it across interests. In this case, the incremental impact is assessed as 14%.

8. Optional: calculate overall impact
   • Drawing on the literature, discussions with experts in Phase 3, the technical advisor or other sources, assign a weight to each outcome based on its estimated contribution to program incremental impacts. The weights assigned to each outcome should sum up to 1.0, or 100%. In the simplest case, weights may be equal across all outcomes.
   • For subject matter experts, calculate the contribution of each incremental impact to the overall impact by multiplying the impact for each outcome by the assigned weight.
   • Calculate the estimated overall impact by summing the contribution of the incremental impacts generated in step (b).
   • Repeat steps (b) and (c) for the other two expert groups.

The calculation described in step 8 provides an estimate for each expert group of their assessment of the program’s contribution to overall impacts in percentage terms. For example, consider a simplified version of the NRCan pilot, with only three outcomes:

• enhancing knowledge in the standards community
• more harmonized codes and standards for alternative fuels
• increased efficiencies in deploying natural gas vehicles

The RIE finds that subject matter experts assess the program’s incremental impact on each outcome as 10%, 15% and 20%, respectively. However, the evaluator might conclude after reviewing the literature and speaking to the experts that for contribution to overall impact, the first two outcomes were equally important, but the third outcome was only half as important as the first two. The weights would then be 40%, 40% and 20%, and the overall impact as estimated by subject matter experts would be 0.4 × 0.1 + 0.4 × 0.15 + 0.2 × 0.2 = 0.14 (or 14%). (For the full NRCan example, see Annex C of NRCan’s Evaluation Report: EcoENERGY for Alternative Fuels Program).

Table 3 provides the assessment of overall impact for a single expert group for a program with four outcomes (Outcomes A to D). Outcome A, whose incremental impact is estimated at 0.14 in Table 2, is assigned a weight of 0.2 in comparison with Outcome B, which is assigned a weight of 0.4. This means that Outcome B is assessed to be twice as important as Outcome A in terms of its contribution to the larger program impact. The overall impact is 0.1625 (or 16%).

9. Optional: verify analysis
   • A useful step in any evaluation is to verify the validity and reliability of the information obtained. Such verification can be done statistically and by comparing the findings of the RIE to other evaluations of the same or similar programs.
10. Report
    • You can now report the findings of the evaluation for each of the direct outcomes and impacts identified. For example, subject matter experts might assess that the program enhanced knowledge within the standards community by 30% more than an alternative delivery option.

Optional: overall impact

An optional step in the analysis is to calculate the overall impact of the program. The simplest method would be to take the mean impact on all outcomes. However, there might also be a justification for weighting impacts differently if some are expected to contribute more to the overall impact than others.

Depending on the question the evaluation is trying to answer, this step may not be necessary. If outcomes are diverse or the evaluation is interested in incremental impacts on specific outcomes, not in general, then incremental impact on each outcome may be what is of interest. This approach also avoids the need to pick a weighting for each outcome, which may be difficult.

Verification

A useful step in any evaluation is to verify the validity and reliability of the information obtained. Since RIE consults three expert groups, it is particularly useful to check whether the three groups are consistent (inter-rater reliability). Cronbach’s alpha is a standard test for internal reliability and can be computed in statistical packages such as SPSS, Stata or R. It is typically not straightforward to do so in the basic Excel package, however.

The evaluator might also check whether RIE estimates are in line with those from other evaluations done of the program or studies of similar programs. Comparing estimates could be done statistically or through a literature review or background research. For example, finding that all previous impact evaluations of a program had found one thing, and that the RIE had found another, would suggest that more research was needed to identify the reason for the difference. Evaluators might start by checking whether the three expert groups varied in whether they disagreed with past findings, and whether the differences were due to methodology, timing or other factors. Past RIEs have typically found that experts from within the program are more optimistic than external ones, which might explain the differences if past evaluations relied more heavily on information from within the program.

Reporting

One of RIE’s strengths is to report a clear and concise summary of what experts assess to be the program’s impact, as well as the potential impact of one or more alternatives. Senior executives and program management are then able to reflect on results in the context of alternatives and discuss options for program improvement directly. Such reflection is particularly important when assessing program effectiveness, but may also be informative when discussing program efficiency relative to possible alternatives.

Providing evaluation readers with this information requires a clear explanation of what information the RIE has provided. In brief, the process outlined above allows the reporting of several key elements:

• a program summary, including the intended and unintended outcomes of the program, which external experts and program stakeholders agree is a reasonable reflection of the program
• a measure of the effect of the program and the counterfactual(s) on each outcome as assessed by the expert groups
• the impact of the program relative to some feasible and plausible alternative(s). that is, what happened due to the program that would not under the alternative
• optional: a measure of the overall impact of the program

Findings are typically best reported as percentages. For example, subject matter experts might assess that the program enhanced knowledge within the standards community by 30% more than an alternative delivery option. Program stakeholders might assess an impact of only 20%, and technical advisors, 25%. Overall, therefore, experts assessed that the program enhanced knowledge within the standards community by 20% to 30% more than the alternative.

If the overall impact has been calculated, for transparency this is also best reported as a range. If the three expert groups assessed overall impacts of 11%, 14% and 16%, the evaluation can report that the program is expected to provide an overall improvement in outcomes of between 11% and 16% more than the alternatives considered. Although it is possible to calculate a single average for the three groups, such a calculation can be misleading if the three groups provided a wide range of assessments. A range is typically preferable.

In some cases, there may be systematic differences between the estimates of each group of experts, which can suggest interesting areas for further analysis or discussion. For example, subject matter experts might be significantly less optimistic about the program’s impacts than internal program stakeholders. Such a perspective might have been missed by an evaluation that did not seek external perspectives and can highlight important issues. Internal stakeholders might be missing some negative unintended consequences, for example, or not be aware of a similar program elsewhere that performed poorly. Equally, internal stakeholders might have access to additional information that supports their optimism but that has not been well communicated or researched. Such findings can be of interest to management and suggest areas for further exploration and evaluation.

Public Health Agency of Canada (PHAC)

Natural Resources Canada (NRCan)

Public Safety Canada (PS)

Summary of departmental experiences

The pilot project was not without challenges. Several departments noted that during the pilot, certain aspects of the evaluation design, and particularly the attempt to fit into the RIE framework, were done somewhat too quickly and with relatively little guidance available. As a result, there was not enough time to get a better understanding and appreciation for some of the RIE steps and requirements.

That said, participants felt that RIE was a good addition to current approaches and methods in evaluation, particularly for ex-ante and mixed-method evaluations, and those supporting program renewal. Quantitative assessments from experts can help add validity and credibility to evaluation findings. Its use as an evaluation methodology may be somewhat limited, however, due to its reliance on expert perception and the potential difficulty in developing a counterfactual in complex contexts. Departments should consider carefully where to apply the approach, as it may not be applicable to all programs.