Mid-term evaluation of the Impact Canada Fellowship Program

[ PDF version ]

Final Report
July 2020

Background

Program context

Impact Canada, announced in Budget 2017, is a unique, first-of-its-kind initiative in which PCO partners with departments in co-designing and delivering policy responses that utilize specific methods such as challenge prizes, innovative finance, behavioural insights and impact measurement. This partnership model was chosen to ensure that as the government moves to scale up and mainstream these new policy tools, they are done in a way that is consistent across government departments and methodologically sound.

Given then novel nature of these approaches, it became quickly apparent in executing on Impact Canada’s mandate, which grew rapidly since inceptions, that the small core team of highly skilled, interdisciplinary team members at the Impact and Innovation Unit (IIU), could not bridge the skills and/or capacity gap across departments in four distinct business lines: behavioural insights, impact measurement, innovative finance, and challenge prize. The skills and capacity gap needed to be addressed in a coordinated way. As such, the Fellowship program was established in 2018 to augment the expertise needed in order to advance Impact Canada’s objectives, and bridge the capacity gap.

The Fellowship program is mandated to recruit specialized talent and skills into the federal public service to support government organizations with building capacity for the practical application of innovative approaches in four key business lines linked to Impact Canada: behavioural insights, innovative finance, impact measurement and challenge prizes, and also upskill the existing workforce.

The IIU leveraged existing staffing flexibilities to reduce time to staff in the recruitment process, identify high calibre individuals with the right technical skills to advance government priorities, and attract new skills and talents to the public service. Rather than recruit for a particular position, the Fellowship program recruits on the skills that can be applied to any given priority work set out by the Government of Canada, in a particular area of expertise. It is important to recognize that at the core of all of Impact Canada’s work, and inherent in applying these specialized skills, are the aspects of problem solving and using an innovative or unconventional policy lens.

The Fellowship program does not have dedicated funding; the salary dollars associated to the program’s design, implementation, and ongoing operations are expensed against the IIU. The host organizations are responsible for reimbursing the IIU the salary dollars of their Impact Canada Fellows for the duration of their placement. The resource complement dedicated to the program’s design, implementation and ongoing operations has fluctuated between 1 to approximately 1.25 full time equivalents (FTE). This was in part due to the fact that management first wanted a proof of concept before staffing up a team. Four stream leads provided technical support guidance to assessment of applicants, and provided ongoing support, coaching and mentoring to Impact Canada Fellows. Since its inception, the Fellowship program has on-boarded 16 fellows, who have been placed in 10 different federal organizations and assigned to work on approximately 18 different projects.

The Fellowship program’s design and delivery is expected to achieve the following outcomes: 

• Yield higher calibre individuals by recruiting based on technical skills;
• Facilitate host organizations’ access to pre-qualified subject matter experts¹ to help build capacity within their organization in four key areas linked to Impact Canada business lines;
• Administer recruitment processes within 90 calendar days resulting in faster and more timely access to pre-screened subject matter experts;
• Provide external subject matter experts greater access to career opportunities within the public service.²

Evaluation objective, scope, methodology and limitations

The objective of this mid-term evaluation was to examine the design and delivery of the Fellowship program to find out if the program was on track to achieve its expected results. The evaluation covered the program’s design and delivery activities since its inception in 2018. Lessons learned from the evaluation are intended to help inform program design and delivery adjustments and improvements, and contribute to broader Government of Canada recruitment efforts.

This evaluation was meant to complement the 2018 Evaluation of the PCO Central Innovation Hub. The evaluation findings will also inform, to the extent possible, the upcoming Treasury Board–mandated Comprehensive Assessment of the Demand for Impact and Innovation Unit’s Innovative Services.

The evaluation was carried out in three phases:

• Planning Phase
  • Developed evaluation planning documents such as the program logic model, Terms of Reference, evaluation framework, as well as data collection and assessment tools (e.g., interview questionnaires and evaluation matrix);
  • Conducted interviews and reviewed relevant documents and files.
• Examination Phase
  • Identified, analyzed, and recorded sufficient information to address evaluation questions;
  • Provided briefings to management on the evaluation progress.
• Reporting Phase
  • Validated the observations with program management;
  • Communicated the observations and drafted a brief report;
  • Identified and presented a set of recommendations for program improvement (see Recommendations).

The following table provides the number and categories of the interviewees:

There are several considerations or limitations to note when reading the evaluation.

• The current evaluation was not designed to opine on whether the people hired through the Fellowship program delivered on the work they were hired to do or whether their projects were delivering higher impact. The answers to these questions will come in the next evaluation planned for 2022.
• The evaluation does not attempt to assess whether the Fellowship program is the right vehicle to attract talent across the public service. This program seeks to demonstrate efficiencies in time to staff; demonstrate that recruiting on technical skills, rather than years of experience, can yield higher caliber candidates; and specifically recruit subject-matter experts from outside the federal public service in four areas linked to Impact Canada: behavioural insights, innovative finance, impact measurement and challenge prizes, with the aim to increase capacity and upskill the existing workforce.
• Given that the program is in its early years, the current small population of interviewees was used to draw conclusions on the stakeholders’ satisfaction of the program’s design and delivery.

Evaluation findings

Finding 1: Recruitment and job advertisements

The program’s recruitment activities have been successful, particularly in terms of the extent and diversity of its reach, as well as its ability to attract the type of individuals it is looking for. For example, since its inception in 2018, the program has run six recruitment campaigns and has received a total of 1,032 applications from across Canada (all 10 provinces and three territories) and several from outside of Canada. Forty six percent of the applicants are estimated to be female and 43 percent to be male (the gender of 11 percent was unknown).

Since the program’s inception, 21 fellows have either been hired or qualified to be hired under the four streams. Out of these 21 fellows, 12 are female and nine are male. At this time, two Fellows have joined the public service permanently, and another is in the process of a deployment to his host department, indicating that departments are receptive to the innovative practices that Fellows are expected to implement. The IIU and Fellowship program manager have assisted with these transitions and have an approach in place to help each Fellow understand their end of Fellowship term options and to explore appropriate linkages with other departments. The majority of the interviewees who have been exposed to the program’s recruitment ads found them different from typical public service job ads. The ads were described to be simple, direct and persuasive enough to encourage some of the applicants who had not previously considered applying for a job within the public service to apply. The use of social media as the primary medium for posting job ads was also described to be an effective outreach strategy that has contributed to the success of the program’s recruitment campaigns.

The Calls for Placement (i.e., the solicitation of departments’ interest for the placement of the fellows) are primarily done through informal networks such as interdepartmental Assistant Deputy Minister and Director/Director General networks. The IIU considers a placement that directly supports the Government’s agenda (e.g., Speech from the Throne, Budget, mandate letter), whether it falls under the auspices of Impact Canada or not. The Fellowship program allows the IIU to continue to provide high level, high quality support to departments, who are utilizing novel outcomes-based policy approaches for the first time, and often require significant guidance and support for prolonged periods of time.

A review of some of the administrative documents indicated that since its inception, the program has received at least 55 official placement requests from 20 different federal organizations, including PCO itself. (One organization submitted nine placement requests, two organizations submitted six requests each, one organization submitted five requests and the rest submitted between one and three requests each.) The majority of the interviewees indicated that the program is not adequately known across the government. As a result, its potential may not be fully exploited.

The establishment of a more formal and widespread approach to Calls for Placement has been proposed to increase the level of awareness of the program, which could result in a higher number of organizations expressing an interest in what the program has to offer.

Finding 2: Program design, delivery and sustainability

The majority of the interviewees viewed the Fellowship program as a unique⁴ recruitment program in the public service (see our benchmarking exercise with four other targeted HR programs in Annex B). The program’s uniqueness was mostly attributed to factors such as the fellows’ employment arrangement (they are employees of PCO, but are placed in other departments); the efficiency of the hiring process (i.e., it was perceived to be faster and more straightforward than other hiring processes, including other traditional or innovative recruitment programs); the candidate assessment process, which was described to be based on level of expertise than years of experience; and the provision of ongoing support and networking opportunities to the fellows by PCO.

While the sample size for the number of the interviewees is small (20 interviewees in total, five of which had direct affiliation with the program), there is a high level of stakeholder satisfaction with the program’s design and delivery, and the program is perceived to be addressing a real need in the federal public service. The program was also found to be contributing to the broader discussion on how to improve and innovate HR practices in the federal public service, particularly by taking to task certain entrenched HR processes and assumptions such as that the public service is not the employer of choice for ambitious, mobile and/or educated professionals, that recruitment processes take too long to complete, that only applicants with recent or significant years of experience can make it through a process, that applicants are mostly interested in indeterminate (permanent) employment, etc.

The Fellowship program’s ability to put in place a streamlined application process, PCO being responsible for the recruitment process, as well as being the primary employer of the fellows were considered as points of strength for the program. However, the program’s resource level was found to be insufficient. As such, the program may not have the capacity to sustain its current level and/or experience future growth. The program was also found to rely heavily on one resource for the program’s design, delivery, and ongoing operations, as well as a small group of individuals for leading program streams. This has been identified as a management risk. The lack of sufficient resources combined with the absence of a succession planning process have been identified as two important factors that limit program growth and make the program vulnerable to any unexpected change in the composition of its management and key staff. That being said, the IIU’s intent would be to scale the Fellowship program, if it were properly resourced.

Finding 3: Program's contribution to the Centre of Expertise for the Impact Canada Initiative

Impact Canada Initiative is described as a Government of Canada-wide effort established to help government departments “accelerate the adoption of innovative approaches to deliver meaningful results to Canadians.”⁵ These efforts are led by a team of experts housed within the Impact and Innovation Unit of the Privy Council Office, referred to as the Impact Canada Initiative Centre of Expertise.

The majority of the interviewees indicated that the Fellowship program contributes to the achievement of the Centre’s objectives primarily by recruiting subject-matter experts into the federal public service, as well as through its promotion of experimentation and outcome-based approaches in developing and implementing policies and programs to advance government priorities.

Finding 4: Onboarding of fellows

The Fellowship program designed and delivers a tailored onboarding curriculum through which the newly recruited Fellows are welcomed and familiarized with certain processes and procedures of the federal public service. A review of relevant documents indicated that a wide range of topics (machinery of government, pay and compensation, performance and learning plans, values and ethics, etc.) are covered during the onboarding sessions. In addition to the program’s tailored onboarding curriculum, Fellows are also encouraged to attend PCO’s general orientation sessions for new employees.

The majority of the interviewees found the onboarding sessions to be very important for a smooth transition of the fellows into the federal public service, particularly given that the majority of them do not have any prior exposure to work in the federal government. At the same time, some of the interviewees indicated that they found the onboarding sessions to be heavily centered on PCO and central agencies. The culture and work environment at PCO and the context for certain topics discussed in the course of the onboarding sessions were described as being very different from the realities of line departments. To mitigate this, some of the interviewees suggested that the host departments be given an opportunity to collaborate in the design and delivery of the onboarding sessions to better incorporate line departments’ perspectives.

Finding 5: Placement and pairing of fellows

The placement and pairing (i.e., assignment of the fellows to departments) is done through an informal and interactive process. According to the existing practices, in response to PCO’s Calls for Placement, departments describe their projects and express their interest for having a fellow from a particular stream to be assigned to their departments. The relevant program manager at PCO identifies the closest match from the pool of candidates based on their expressed interest and area of expertise. After going back and forth several times with both the candidates and the host departments, and in some cases, after providing the two parties with the opportunity to meet in person to discuss their needs and interests further, the best fit is determined by the PCO Stream Lead.

The majority of the stakeholders expressed their satisfaction with the existing practices albeit in some cases with some caveats. In addition, it seems that with the exception of a few cases, the process has worked well as the majority of both the fellows and host managers were satisfied with the placements. Nevertheless, departments have indicated that the existing placement processes limits their choice somewhat and have expressed interest in receiving a list of pre-qualified subject matter experts to select from. PCO needs to ensure greater awareness of the program and its services, including its discretion vis-à-vis selection of best fit and link to PCO’s Impact Canada Centre of Expertise, as well as the approach applied for pairing subject matters to host organizations.

There is a perception that some host managers did not get their first choice candidate, perhaps, as a result of the way PCO presented projects and departments to potential candidates. PCO may wish to consider including host managers in the final stage of placement discussions to reduce misconceptions or misunderstandings vis-à-vis placement opportunities. Including host managers in placement discussions would allow host organizations to better characterize their project work and departmental context, and perhaps better support potential candidates in their decision-making.

There have been instances where after the placement of a fellow in a particular department, it was found that the host department had a misconception of the Fellowship program and the type of expertise that the fellow was intended to bring to the department. In addition, soon after starting their placement, some fellows encountered a total lack of preparedness on the part of their host department. In some cases, this resulted in a delay in starting the work, and in certain cases the host manager did not seem to have a concrete plan for the fellow’s integration. To avoid and/or minimize such situations, it was suggested that PCO should better scrutinize the departments’ expressions of interest to minimize misconceptions and to ensure preparedness. A review of the existing placement processes also points to PCO stream leads having considerable discretion in determining the best fit. Although the process has so far worked well, there is a risk that any unexpected change in the composition of the team, particularly the stream leads, may negatively impact the effectiveness of the placement processes. As such, it would be prudent for the program management to introduce certain checks and balances to ensure consistency in determining the best fit in the future.

Finding 6: Achievement of outcomes

A majority of the interviewees indicated that the program has been successful in attracting and bringing high calibre individuals into the government. (High calibre is defined as individuals who perform beyond expectation on a consistent basis.)

By the same token, the program has been able to facilitate departments’ access to prequalified subject matter experts in the four business lines (i.e., innovative financing approaches, challenge prizes, impact measurement methodologies, and behavioral insights). In most cases, such expertise did not exist in these departments. Although it is too early in the life cycle of the program to assess the extent to which the fellows have been able to help build capacity within their host departments in these four areas, there is a sense of acknowledgment and satisfaction on the part of the host departments with the performance of their respective fellows.

Document review indicated that the program has been able to meet its established 90 calendar day target for pre-screening the candidates in four out of six recruitment campaigns. The shortest processing time was 42 days and the longest 145 days.

The majority of the stakeholders have acknowledged the program’s success in establishing a streamlined recruitment process, particularly up to the formation of a pool. However, some of the stakeholders observed that the rest of the process (that is from the formation of the pool to securing a placement) is still long.

Although there was a recognition that some of the barriers impeding efficiencies in the process could be systemic, it was proposed that PCO continue to explore ways to foster greater program efficiencies in areas for which it has direct control over and work with the appropriate stakeholders to address systemic barriers that prevent bringing greater efficiencies to its program. 

The program has been deemed successful in providing the fellows with greater access to career opportunities within the federal public service. In fact, some of the fellows have already been offered longer-term employment opportunities within their respective host organizations. However, several of the fellows and host managers pointed out the absence of a consistent government-wide approach to retaining fellows who are interested in joining the federal public service on a long-term basis. The existing approach was described as being on a case by case basis that is, to a large extent, dependent on the ability and willingness of a host manager to justify a non-competitive appointment. 

Similar to other staffing options provided for managers to bridge students or appoint casual/term employees into positions in the federal public service, the Fellowship program is also in need of adopting a government-wide mechanism to facilitate the integration of the interested fellows in the federal public service in order to maintain its appeal. 

Finding 7: Other considerations

It is important to note that although the program is known for recruiting subject-matter experts from outside the federal public service, fellowship opportunities are also available to existing and/or former public servants. In fact, over the course of the past 6 recruitment campaigns, several existing public servants have participated in the recruitment processes. The Fellowship program manager and the PCO Human Resource Advisor have discussed the various considerations for onboarding an existing public servant into the Fellowship program. Public servants who pre-qualify as subject matter experts would be presented with their onboarding options on a case-by-case basis. Mechanisms are in place to ensure employee rights are respected, as well as help ensure continued employment in the public service (i.e., that there are no breaks in service).

PCO should better publicize this option for public servants, for instance, by including a reference to the consideration of existing public servants to their frequently asked question repertoire.

Conclusion of findings

The program has been successful in its recruitment efforts, particularly, in terms of the extent and diversity of its reach, as well as attracting the type of individuals it is looking for. In addition, there is a high level of stakeholder satisfaction with the program’s design and delivery, especially with the fact that PCO does the recruitment and the fellows are considered PCO employees, and as such, they are well supported by their respective peer networks and PCO managers/stream leads. 

The program has positively contributed to the achievement of its expected outcomes, particularly with respect to recruiting high calibre individuals and facilitating host organizations’ access to pre-qualified subject matter experts. It has also met in four out of six recruitment campaigns its 90 day target for processing applications; and it has provided the fellows with a greater access to employment opportunities within the public service. Having said that, the program is also facing a number of challenges that, from an evaluation perspective, have the potential to negatively impact its efficiency, effectiveness and appeal if left unaddressed. These challenges include an insufficient resource level; the absence of consistent, government-wide processes for calls for placement and the provision of long-term employment to interested high performing fellows; and a lack of clarity surrounding existing public servants’ participation in the program.

Given these findings, the evaluation has put forward several recommendations in the spirit of continuous improvement. These improvements will also inform the upcoming comprehensive assessment of the Impact and Innovation Unit’s innovative services in 2022.

Recommendations

The evaluation recommends that the Assistant Secretary to the Cabinet, Impact and Innovation Unit consider:

1. Establishing a formal government-wide Call for Placement process to ensure there is a greater awareness of the program and its services.
2. Putting in place a succession planning process, such as a cross-pollination opportunity, to ensure continuity of the quality and level of expertise the program provides to both fellows and host departments in the event of any unexpected changes in the composition of its key staff.
3. Exploring options to further support host departments in the design and delivery of the tailored onboarding sessions.
4. Introducing further checks and balances to the existing process to determine the best fit for the fellows’ placement and pairing.
5. Bringing observations vis-à-vis systemic barriers to the attention of the appropriate stakeholders, including the PSC and OCHRO, to further highlight impediments across the system to improve government-wide process efficiencies.
6. Promoting flexibilities granted in the Public Service Employment Act for offering long-term employment in the federal public service to interested, high-performing Fellows.
7. Clarifying and better publicizing the possibility that existing public servants are permitted to participate in the Fellowship program.

Annex A: Management response and action plan

Annex B: Comparative study of the Fellowship program against four targeted HR programs

The four targeted programs in this benchmarking exercise were: Recruitment Policy Leaders, Career Marketplace, Talent Could, and Free Agents.

Annex C: Program logic model