Archived - Neutral Assessment of Finance Canada's Evaluation Function

Presented to the
Audit and Evaluation Committee
December 9, 2013

Executive Summary

Validation / Neutral Assessment Report

Finance Canada engaged Public Accountability Consulting Inc. to validate a self-assessment of the evaluation function at Finance Canada in order to meet the Treasury Board (TB) Policy on Evaluation requirement for a neutral assessment.

We reviewed the self-assessment, reviewed documents and evaluation studies, conducted interviews and a focus group with departmental evaluators. We rated the department with regard to the requirements of the TB Policy on Evaluation.

We concur with the conclusions in the self-assessment report and are of the view that: The Department of Finance and the evaluation function is in general conformance with the requirements of the TB Policy on Evaluation, and associated directives and standards.

We have made two recommendations for improvement.

Finance Canada engaged Public Accountability Consulting Inc. (PAC) to validate a self-assessment of the evaluation function at Finance Canada in order to meet the requirement for a neutral assessment. This document presents our results.

The Treasury Board (TB) Policy on Evaluation requires deputy heads to establish a robust, neutral evaluation function and to ensure that their department adheres to the Policy on Evaluation and its supporting directive and standard. As part of monitoring compliance to this policy, a neutral assessment of the evaluation function must occur at a minimum every five years. This was the first neutral assessment conducted by Finance Canada and it had to be finalized by March 14, 2014.

The purpose of the validation / neutral assessment was to:

1. Assess conformity/compliance of Finance Canada's evaluation activities against the TB Policy on Evaluation, associated directives and standards; and
2. In addition to assessing compliance, assess management practices and identify opportunities and provide recommendations for improvement.

PAC was engaged to validate a self-assessment that was prepared by the evaluation function at Finance Canada. To complete this validation, we designed a customized approach and templates to conduct this assignment. To preserve the independence of this validation, the Project Authority for this assignment was the Senior Director, Corporate Planning at Finance Canada.

PAC held initial meetings with the Project Authority and the Head of Evaluation during August 2013, to discuss the timing and approach of this project and to discuss the tests that would be used.

At that time there was no published Treasury Board of Canada Secretariat (TBS) guidance for the conduct of neutral assessments. Draft guidance was in preparation and this neutral assessment is fully consistent with the draft TBS guidance that had been issued. Since the work of the evaluation function aims to implement the TB Policy on Evaluation and the more detailed elements of the associated directives and standards, the validation of the self-assessment was conducted using the areas of responsibility for deputy heads that are set out in the Policy on Evaluation; i.e. governance, quality of evaluations, use of evaluations, and coverage of the evaluation universe. These are the same areas that are used by TBS in conducting its assessments of the evaluation area of management of the Management Accountability Framework (MAF).

During August and September 2013, the PAC neutral assessment team reviewed the detailed self-assessment completed by the evaluation function, gathered additional background information, reviewed the evaluation studies published to the Finance Canada website as well as key documents such as the 5-year evaluation plan. The PAC team selected senior executives to be interviewed and finalized planning and administrative arrangements. During the field-work phase, we assessed documents, TBS MAF reports, evaluation planning processes, and the evaluation tools and methodologies in use at Finance Canada. We reviewed evaluation engagement and staff management processes and evaluation files. Interviews were conducted with the three Audit and Evaluation Committee external members, the Deputy Minister, the Associate Deputy Minister, senior executives of two branches that were recently evaluated, and the Head of Evaluation. A focus group was conducted with evaluation staff. The PAC team examined two recent evaluations in more depth (Evaluation of the Tax Policy Branch; and Evaluation of Communications and Consultations Branch), including a review of the working papers, and client surveys prepared at the completion of these two evaluations.

All statements in the self-assessment were reviewed and challenged against the evidence that was collected. We rated the department using summary tables with regard to the requirements of the TB Policy on Evaluation. Once our analysis was complete, we prepared our independent validator statement and completed our neutral assessment report.

(Neutral Assessment of the Evaluation Function)

The Validator was engaged to conduct an independent validation of the Department of Finance Canada's self-assessment of the evaluation function. The objective of the validation was to verify the assertions made in the self-assessment report concerning:

The Department of Finance Canada and the evaluation function's conformity to the requirements of the Treasury Board Policy on Evaluation , and associated directives and standards.

In addition, the validation assessed management practices to identify opportunities and provide recommendations for improvement.

In acting as Validator, we are fully independent of the organization and have the necessary knowledge and skills to undertake this engagement. The validation, conducted from August to October 2013, consisted primarily of a review and testing of the contents, procedures and results of the self-assessment, as well as a rating of the department against the TB Policy on Evaluation requirements.

In addition, interviews were conducted with the Deputy Minister, Associate Deputy Minister, the Chair and External members of the departmental Audit and Evaluation Committee, selected senior executives, and the Head of Evaluation. A focus group was conducted with evaluation managers and staff.

We concur with the conclusions in the self-assessment report and are of the view that:

The Department of Finance and the evaluation function is in general conformance* with the requirements of the Treasury Board Policy on Evaluation, and associated directives and standards.

*Conformance to the TB Policy on Evaluation requirements is evaluated using the following scale:

"Generally Conforms" means there is no material deficiency, although there may be some minor deficiencies.

"Partially Conforms" means there is one material deficiency, and there may be minor deficiencies.

"Does Not Conform" means that there is more than one major deficiency in practice and these deficiencies are judged to be so significant as to seriously impair or preclude the evaluation function from performing adequately in all or in significant areas of its responsibilities.

The evaluation function is highly-regarded by departmental executives for its credible, timely, neutral, and well-documented work. The staff is experienced and knowledgeable regarding the programs and policies of the department and there is relatively low turnover, which allows for continuity in the conduct of projects and a relatively high productivity of the group.

The evaluation function has displayed a significant degree of innovation in the practices it has adopted.

The function has adapted the standard evaluation methods to allow practical and pragmatic assessment of a policy department that delivers very few traditional government "programs".

The function has recently taken steps that aim to complete assignments in a 6-month timeframe.

Evaluation reports are generally relatively short and to-the-point and this is most appreciated by departmental managers.

There is a rigorous follow-up system in place to ensure that management action plans are implemented in a timely manner.

The timing of data collections aims to minimize the time required from Finance management during the most intense period of budget consultations and budget preparation.

While the overall conclusion of this neutral assessment is that the evaluation function is in general conformance with the requirements of the TB Policy on Evaluation, the neutral assessment team has made a number of best practices observations and recommendations that are intended to build on this solid foundation.

The observations that follow cover the following subjects:

Governance: The existing Audit and Evaluation Committee format is working well. The pre-meeting format is a productive innovation. Strong Deputy Minister support has helped the function become established. One committee that provides advice on both evaluation and internal audit has facilitated planning and coordination of the work of these functions. The neutrality of the Head of Evaluation is enhanced due to the direct reporting relationship to the Deputy Minister.

Quality of evaluations: Standard techniques have been well adapted to the evaluation of a policy department. Staff is professional and studies have met high quality standards.

Use of evaluations: The evaluation work to date has been useful for departmental management. Careful planning and consultations with the branch at the start of each study are a worthwhile investment. Brevity and a focus on shorter reports with less text about the non-controversial issues (such as the continued relevance of the work of the branches), are highly-valued. Continuous follow-up to track implementation of management action plans is highly-effective.

Coverage of the evaluation universe: The 5-year plan aims for 100% coverage and this is in full compliance with the TB Policy on Evaluation. In the second round of studies, a more focused approach may provide higher value. A focus on highest risk issues and priorities is likely to be beneficial.

The first round of evaluations provided coverage of the core TB evaluation issues related to both relevance and performance; this is in full compliance with the TB Policy on Evaluation. The core issues related to relevance include the continued need for the program; alignment with government priorities; and alignment with federal roles and responsibilities. As these issues were assessed during the first round of the Finance evaluations, and since the relevance of Finance's policy advice functions has been demonstrated, unless there has been a significant change from what was examined, the level of effort to examine these issues a second time can be reduced. This is consistent with the flexibility provided in the TB Directive on the Evaluation Function with respect to the coverage of core issues. This is expected to enable the Finance evaluation function to focus on the key performance issues and produce evaluations that are of maximum use to senior managers, given the importance and fundamental requirements in the department for policy advice.

The internal self-assessment by officials of the Finance evaluation function considers the capacity to supply evaluation information, including: appropriateness of structure and resourcing of the evaluation function; the management of the evaluation function; evaluation planning and coverage; evaluation processes and activities; evaluation planning and design; evaluation project management; and evaluation reporting. The self-assessment then considers the capacity to demand and use evaluation information, including: evaluation utilization at the agency level and at the level of individual evaluations; and meeting Deputy Minister needs. The final section of the self-assessment considers the enabling environment to support evaluation (its conduct and use) at Finance, including the existence of a senior level committee (the Audit and Evaluation Committee) to provide the necessary support and oversight of the evaluation function; access to information required for evaluation; and responsibilities of program managers.

We confirm that:

• we have reviewed the standards and ratings and the comments made in the self-assessment;
• the self-assessment covers the full set of requirements set by the TB Policy on Evaluation, and associated directives and standards;
• we concur with the ratings noted in the self-assessment and that the comments provide adequate evidence to support the ratings; and,
• the comments noted in the self-assessment were consistent with our independent findings based on the review of documentation, review of reports posted to the Finance website, in depth review of two recent evaluation reports, and the interviews and focus group we conducted.

Recommendations:

As the evaluation function is operating in full conformance with the TB Policy on Evaluation, these recommendations are provided for the consideration of management of the function as a potential improvement to what is already working well.

Recommendation 1: For the second round of its evaluations, we recommend that Finance look at preparing shorter reports, targeted to the issues of highest risk, priority and interest regarding the subject matter of the report.

Recommendation 2: We recommend that the Finance evaluation function continue to take advantage of the flexibility available for departments under the TB Policy on Evaluation to reduce the level of effort during evaluation projects related to the core issues of relevance.

Management response to recommendations 1 and 2: The Head of Evaluation agrees and both recommendations are currently being implemented. The two recommendations are expected to be fully implemented by March 31, 2015.

The neutral assessment team wishes to express its appreciation for the cooperation and assistance afforded by the Deputy Minister, Associate Deputy Minister, Audit and Evaluation Committee members, Finance Canada senior management, and the Head of Evaluation and evaluation staff.