Guidance on the Neutral Assessment of the Departmental Evaluation Function

Table of Contents

• Acknowledgements
• 1.0 Introduction
• 2.0 Purpose of Neutral Assessments
• 3.0 Roles of Heads of Evaluation and of Departmental Evaluation Committees in Neutral Assessments
• 4.0 Developing Assessment Approach and Scope
• 5.0 Defining the Assessment Period
• 6.0 Developing the Assessment Framework
• 7.0 Achieving a Neutral Assessment
• 8.0 Reporting Considerations
• 9.0 References

Acknowledgements

This document was developed by the Centre of Excellence for Evaluation at the Treasury Board of Canada Secretariat. The Centre would like to thank the departments and agencies that reviewed and provided comments on draft versions.

1.0 Introduction

Deputy heads are responsible for monitoring compliance with the Policy on Evaluation in their departments to ensure its effective implementation. Under section 7.1 of the Policy, "[t]hey are responsible for ensuring that a neutral assessment of their departmental evaluation function is conducted at a minimum of once every five years."

The Treasury Board of Canada Secretariat has developed this guidance to help departments design and conduct these assessments. Each department can design its assessment in the manner that best meets its information needs and those of its deputy head.

To comply with the Policy, each large department and agency must complete and approve its first neutral assessment by March 31, 2014. Each subsequent neutral assessment must be approved no later than five years after the previous one.^{Footnote 1}

Under section 2.2 of the Policy on Evaluation, the requirement for conducting a neutral assessment is deferred for small departments and agencies, defined as organizations having less than $300 million in Annual Reference Level and revenues credited to the vote.

2.0 Purpose of Neutral Assessments

The main purpose of neutral assessments is to support deputy heads in fulfilling their responsibility for monitoring compliance with the Policy on Evaluation to ensure its effective implementation. Departments can use the information generated through these assessments to target improvements in their evaluation function and to establish a baseline for tracking progress or changes.

3.0 Roles of Heads of Evaluation and of Departmental Evaluation Committees in Neutral Assessments

As the lead for the evaluation function in the department, the head of evaluation informs the deputy head and the Departmental Evaluation Committee about the requirement to conduct a neutral assessment at least once every five years and the deadline for completing the next neutral assessment. The head of evaluation or the Departmental Evaluation Committee may present the deputy head with proposals that outline possible approaches to conducting the neutral assessment, as well as the strengths and weaknesses of each. The proposals may include the following:

• Which purpose(s) to target through the assessment;
• The assessment approach, scope and period; the elements that will be assessed; the methods and data that will be used to assess them; and the sources of that data;
• The measures that will be taken to ensure that the assessment is neutral (e.g., considerations relating to governance and oversight of the assessment and to the appropriate involvement of the evaluation unit itself); and
• The approach for reporting on the assessment.

The head of evaluation, the evaluation unit and the Departmental Evaluation Committee are the key players in the departmental evaluation function, and departments are advised to give careful consideration to the extent and form of their involvement in the assessment to ensure that the assessment is neutral.

4.0 Developing Assessment Approach and Scope

The assessments required under section 7.1 of the Policy on Evaluation cover the evaluation function—not just the evaluation unit or branch.

Departments are advised to consider all elements of their evaluation function as possible assessment elements, but they have the flexibility to choose an assessment approach and scope that will be useful and meet their needs. Assessment elements can include the Departmental Evaluation Committee and its role and responsibilities as described in the Policy on Evaluation; they also include program managers and their roles and responsibilities in relation to performance measurement as an element that supports evaluation.

Once departments have decided on an assessment approach and scope, departments can document it in an assessment framework that specifies the methods, data sources and analyses that will be used.

The following are some possible approaches to conducting the neutral assessment:

• Comprehensive, requirement-by-requirement assessment of compliance

  This approach involves an examination of a comprehensive list of policy requirements (e.g., those set out in the Policy on Evaluation, the Directive on the Evaluation Function and the Standard on Evaluation for the Government of Canada) in order to draw conclusions about compliance with each individual requirement and about the effectiveness of implementation.

• Assessment by theme

  This approach involves an examination of the evaluation function by theme (e.g., governance, management practices, evaluation practices, and evaluation use) in order to draw conclusions about policy compliance and effectiveness of implementation in each theme. Each theme would be examined through a grouping of related policy requirements and, optionally, any related organizational procedures, processes and practices.

• Values-based assessment

  This approach involves an examination of the evaluation function with respect to how well it demonstrates key values or qualities that are important for achieving the expected results of the Policy on Evaluation. Values could include credibility, independence and the usefulness of evaluation products. The assessment could align individual policy requirements and, optionally, any related organizational procedures, processes and practices, with the key values they support. The assessment would draw conclusions about the extent to which the key values were demonstrated, based on compliance with policy requirements and on effective implementation of the Policy.

• Risk-based and targeted assessment

  These approaches take contextual and risk factors into consideration to focus the assessment on selected aspects of the evaluation function, for example:

  • The approach may involve examining all policy requirements to some degree but examining certain requirements in more detail.
  • The approach may involve examining a subset of policy requirements in order to draw broader conclusions about policy compliance. Conclusions could be organized by theme or by principle or value.
  • The choice of individual policy requirements, themes, evaluation principles or values to examine, as well as the depth of the examination could be based on known or perceived strengths and weaknesses of a department's evaluation function or on the department's priorities for improvement.
  • In rare situations, such as when a department is undergoing significant organizational change or when a department has a less mature evaluation function, the approach could focus more on the capacity for producing evaluation information and less on assessing the use of that information for decision making.

5.0 Defining the Assessment Period

Although the Policy on Evaluation requires that a neutral assessment be conducted at least once every five years, it does not specify the period or the number of years to be covered in the assessment.

Departments can choose a period that meets their information needs and those of the deputy head, and contextual factors can be taken into account.

Departments could choose to assess the entire five-year period since the previous neutral assessment was completed, or they could decide to focus on the present-day situation for some aspects of the assessment and look at all five years for other aspects (e.g., those relating to evaluation coverage).

6.0 Developing the Assessment Framework

Once a department has chosen the approach, scope and period for the neutral assessment, it can develop a framework that specifies the elements to be examined (e.g., policy requirements, themes, values), the data collection methods, the data sources, and the planned analyses.

Departments can tailor the assessment framework to their needs. They can include any elements they deem useful and can determine how those elements will be examined. They can also determine the level of effort needed to conduct an assessment that will be sufficiently robust and reliable. A previous neutral assessment of the evaluation function can help in calibrating subsequent neutral assessments, for example, through changing the focus or scope.

Departments can collect primary data, secondary data, or both, and they can use a wide variety of methods to collect it, for example, document reviews, interviews or consultations, focus groups, case studies of individual evaluation projects; and interdepartmental comparisons of evaluation structures, processes or procedures.

Possible secondary data sources include the evaluation unit's performance measurement strategy, post-evaluation client surveys or consultations, management reviews or studies, other assessments, departmental evaluation plans, and annual reports on the state of performance measurement that are prepared by the evaluation unit or the head of evaluation.

7.0 Achieving a Neutral Assessment

To be neutral, an assessment should be governed, designed, conducted and reported impartially. The people who carry out neutral assessments should not allow official, professional, personal or financial relationships or interests to influence or limit the scope and rigour of the assessment, to limit disclosure, or to weaken or bias the findings. They should not allow preconceived ideas, prejudices or biases to affect the analysis of the data collected in the assessment; the development of the findings and conclusions, and the tone and content of the assessment report.

Deputy heads can determine the appropriate means for ensuring the neutrality of the assessment. For example, the assessment could be overseen or led and conducted by parties external to the departmental evaluation function (e.g., a separate organizational unit or a steering committee). Alternatively, if the deputy head considers it appropriate, the assessment could be led by the head of evaluation or by the Departmental Evaluation Committee and could include a mechanism for scrutiny and validation by peers or by experts external to the departmental evaluation function at key stages of the assessment process, for example, during the contracting process (if applicable), during the actual assessment work and during the review of draft and final assessment products.

Deputy heads are responsible for neutral assessments. It is therefore advisable for them to approve the approach, the assessment framework and the final assessment report.

8.0 Reporting Considerations

Departments can choose any reporting format that meets their needs, but they are advised to consider the immediate needs of readers, the needs of the people who will follow up on the assessment, and the needs of the designers of subsequent assessments.

9.0 References

• United Nations Evaluation Group, Development Assistance Committee, "Framework for Professional Peer Reviews"