Evaluation of the Provincial Nominee Program

2. Evaluation description and methodology

2.1. Evaluation objectives

In accordance with the Treasury Board (TB) Policy on Evaluation (2009), the objective of this evaluation was to examine the relevance and performance of Citizenship and Immigration Canada’s (CIC) Provincial Nominee Program (PNP). Program relevance was assessed in terms of

1. continued need;
2. alignment with government objectives and priorities; and,
3. consistency with respect to federal roles and responsibilities.

Program performance was assessed by examining results in terms of

1. effectiveness and
2. efficiency and economy.

Thirteen evaluation questions were developed by CIC to respond to the two overall evaluation themes (Table 2-1).

The evaluation also gathered information for a program profile that provides a description of the operation of the PNP for each jurisdiction in which it is delivered. The profile briefly describes the individual streams, monitoring and accountability mechanisms, and a description of the approach used by provinces and territories to determine their need for provincial nominees (PNs). Thus by providing an overview of each PN program, it allows a better national understanding of how the PNP operates.

Table 2-1: Evaluation issues and questions

2.2. Methodology

The timeframe of the evaluation covered the 2005 – 2009 period, however, part of the analysis used the Longitudinal Immigration Database (IMDB), which includes PNs landed from 2000 – 2008.

Data collection took place between November 2010 and May 2011; the instruments are presented in a technical annex, available under separate cover.

The evaluation data provided about the PN population is always on Principal Applicants, unless otherwise noted. Similarly, for comparison purposes, data on other economic programs is always reported for Principal Applicants. Additionally, data on spouses and dependants is out of scope for this evaluation.

2.2.1. Document review

The document review provided evidence for most evaluation questions. The following types of key documents were reviewed:

• Program and background information, including Provincial/Territorial (PT) overviews and profiles and links to relevant web sites;
• Minutes of the Economic Working Group (EWG) meetings from 2005 to 2010;
• Information on other federal economic immigration programs, including Federal Skilled Worker Program (FSWP), Temporary Foreign Worker Program (TFWP), and the Quebec Skilled Worker Program;^{Footnote 9} and
• Other PNP evaluations (conducted at the PT level), analyses/reviews, including Office of the Auditor General (OAG) and PT audits and other research papers.

A list of documents is presented in Appendix B under the technical documents appendix, included under separate cover.

2.2.2. Key informant interviews

Key informant interviews were conducted to respond to all evaluation questions. A large proportion of these interviews were conducted in person, many as part of a site visit to each jurisdiction administering a provincial nominee program^{Footnote 10}.

A total of 132 interviews were conducted with 197 individuals, according to the following breakdown by respondent type (see Table 2-2):

• CIC National Headquarters personnel (n=5 interviews with 6 individuals), including program staff and senior managers at CIC;
• CIC regional and local office staff and managers (n=13 interviews with 17 individuals), including representation from all four regions covered by PNP, as well as Quebec;
• Representatives from PT governments (n=41 interviews with 83 individuals), including individuals in each PT (where a mix of interviewees who could speak from the immigration perspective as well as the labour force strategy perspective was sought), and one representative from the province of Quebec^{Footnote 11} who was asked to share descriptive information about the selection of economic immigrants destined to Quebec;
• Representatives from external stakeholder groups at the national or PT level (n=53 interviews with 71 individuals), including relevant industry associations, labour federations, sector councils, unions, national and PT regulatory bodies, employer associations, as well as some interviews with immigration agents, intermediaries, lawyers and/or immigration consultants; and
• Representatives from CIC visa offices (n=20), comprised entirely of Immigration Program Managers (IPMs), conducted after the survey of visa offices (see below).

Table 2-2: Key informant interviews by respondent type

2.2.3. Administrative data review/statistical analysis

The main source of administrative data was the Field Operations Support System (FOSS) and the Longitudinal Immigration Database (IMDB).

FOSS is a CIC system that contains landing information on immigrants entering Canada. The analysis of the FOSS data provided detailed profile information on the 33,723 Principal Applicants (PAs) under the PNP for landing years 2005 to 2009. This analysis could be conducted by nominating PT, but not by program stream since stream information is not captured in the FOSS database^{Footnote 12}.

IMDB is a database that is managed by Statistics Canada on behalf of a federal-provincial consortium led by CIC. This database links records from FOSS to Canada Revenue Agency (CRA) tax files (mainly T1 data). It is a tax filer database that contains information on all landed immigrants in Canada since 1980 who have filed at least one tax return. The analysis for this evaluation was based on all PNP PAs for the landing years 2000 to 2008^{Footnote 13}, which totals 21,715.

Table 2-3: Number and percentage of PN PAs present in the IMDB for landing years 2000-2008, by PT

The IMDB was used to assess a number of evaluation issues related to economic outcomes, retention, and mobility of PNs. The analysis considered employment earnings, self-employment, employment insurance and social assistance benefits. The analysis of the outcomes for PAs was based on extensive bivariate tables as well as regression modelling. For the purpose of the analysis, all earnings were adjusted using the Consumer Price Index (CPI) to account for inflation^{Footnote 15}. This allows comparison of earnings across different years.

2.2.4. Focus groups with provincial nominees

Focus groups were conducted with PNs in order to obtain an in-depth understanding of client outcomes. In all, 14 focus groups were conducted with a total of 122 PNs, as outlined in Table 2-4.

Table 2-4: Focus groups conducted by jurisdiction

All focus groups except one were organized by the PT representative(s). Each group lasted about two hours and participants were provided with a $50 honorarium for their participation and to offset any costs they may have incurred to participate in the focus group.

2.2.5. Employer interviews

In all, 67 interviews were conducted with employers who were identified with the help of the PTs. They included 14 conducted in-person during site visits and 53 conducted by telephone. Of the 67, ten interviews were conducted with “large” employers, defined as those with more than 10,000 employees in Canada. Table 2-5 presents the breakdown of employers interviewed by jurisdiction.

Table 2-5: Employer interviews conducted by jurisdiction

* Note that one of these interviews was, in fact, a focus group with five employers.

Employers in a wide variety of sectors of the economy were interviewed, with the largest numbers being:

• health care and social assistance (n=11);
• manufacturing, machining and fabrication (n=8);
• agriculture, forestry, fishing and hunting (n=8);
• professional, scientific and technical services (n=6);
• accommodation and food services (n=6); and
• meat processing and packaging (n=6).

Employers interviewed hired PNs at a wide variety of skill levels, including Canadian National Occupational Classification^{Footnote 18} (NOC) level A where occupations usually require university education (n=27), NOC level B where occupations usually require college education or skilled apprenticeship (n=25), NOC level C where occupations usually require secondary school and/or occupation-specific training (n=21) and NOC level D where occupations usually require a short demonstration or on-the-job training (n=14). Only five employers interviewed had hired PNs at NOC level 0 which are management occupations.

2.2.6. Survey of PNs

A survey of PNs was conducted between April and June, 2011. Prior to the survey, a letter from CIC seeking consent to participate was sent to all PAs (about 33,000) with landing dates from 2005 to 2009. Of the 4,894 forms returned, 4,316 contained signed consent. Of those, 4,065 provided usable contact information.

The survey was administered through a hybrid of online and telephone interviews; all 2,357 PAs who provided an e-mail address were sent an e-mail inviting them to participate in an online survey, while other respondents were reached by telephone. In total, there were 2,655 survey completions (1,932 online completions and 723 completed by telephone). For questions where most respondents provided an answer, the sampling error would be approximately +/- 1.9% using a 95% confidence interval.

The response rate was calculated based on the number of survey completions within the given timeframe for the survey out of the total (i.e.: those who consented and provided valid contact information). Using this approach, the response rate for the survey of PNs was 65%.

Table 2-6: Number and percentage of completed surveys, by PT

The interview could be completed in English or French or one of five languages chosen because they were the top five languages spoken by PNs (according to FOSS). The distribution of respondents by these languages is provided in Table 2-7.

Table 2-7: Language of completed surveys

The survey data was weighted based on the FOSS data profile for the PN population to reflect the known population characteristics. The weighting was conducted for the nominating province, year of landing, skill type of intended occupation, age, education, and language. A comparison of the profile of the weighted survey respondents on these variables to the profile of the PN population in the FOSS database is provided in the technical appendices. Also, the NOC skill level of the intended occupation of the PNs captured in FOSS was compared to the NOC for the actual employment obtained as described in the survey.

2.2.7. Survey of visa offices

A survey of managers and visa officers at CVOAs was conducted between April and June 2011. Contacts were provided for the 48 visa offices that process PN applications. An e-mail invitation was sent to the Immigration Program Managers (IPMs) who were requested to forward the invitation to visa officers at their location who process PNP applications. In total, there were 62 respondents to the survey from 36 offices. This represents a 75% coverage rate across offices (36 of 48). It is not possible to calculate a response rate for the population of CIC visa officers involved in the processing of PN applications. The total number of individuals sent an invitation was not known, as the managers distributed the surveys to an undisclosed number of visa officers. Moreover, the total number of visa officers who process PN applications across the network is not known.

2.3. Scope and limitations

Given that this is a federal evaluation, the intention was not to evaluate specific PT PN programs.

Provinces and territories are responsible for the monitoring of their programs and are best placed to assess the unique situations under which their programs operate. Based on the role CIC has in the PNP, this federal evaluation focused on assessing the PNP from a nation-wide perspective. It would have been challenging for a federal evaluation to analyze the programs at the PT level, and yet provide a national picture of the PNP.

It is important to note that the PT PN programs vary in terms of maturity and the strategies based on specific labour market needs, with differing external factors and potential influences; thus the results across PTs may not always be comparable and must be considered in the appropriate context. This evaluation includes provincial profiles to depict as much as possible the specificities of the program operations within each PT.

The evaluation focuses on principal applicants and does not cover spouses and dependants.

PTs nominate candidates to respond to specific needs, and as such, the evaluation focused on how candidates selected on those grounds fared in Canada. While recognizing that spouses and dependants have an impact on the country and PTs where they settle, the unit of analysis had to be on the principal applicants as no information was available in the IMDB on household income, impeding the assessment of economic outcomes of PNs from a family perspective within the evaluation.

There are several challenges and limitations that should be considered when reviewing the evaluation results. Those limitations, their possible impacts on the analysis, and mitigation steps are discussed below.

Given the nature of the population and program under study, finding an appropriate population and program against which to compare is difficult.

Identifying a control group against which to compare outcomes of the PN population is difficult. Canadian-born population is a group that could have been used for comparison purposes; however, given that Canadian-borns are at different stages of their careers, while PNs are just entering the labour market makes the comparison unequal. Educational and occupational distribution of those two groups may also differ, making any comparison difficult. In addition, PNs do not benefit from the same types of social networks and have a different degree of local labour market knowledge. Comparison of PN outcomes across provinces was done, while acknowledging that differences could not necessarily be attributed to the different PN programs. Economic conditions vary by province, as well as the occupational distribution of PNs admitted to each province. Where appropriate, the evaluation compared outcomes of PNs to those of other federal economic categories. Comparison to other economic categories, in terms of program operating costs, was limited by the fact that part of the PNP processing is done at the PT level, as opposed to other immigration programs where all the processing is done by CIC.

Information by PNP stream is only collected by PTs, and not in a consistent manner across PTs, limiting the evaluation’s ability to present a comprehensive picture of PNs by stream.

There were no mechanisms nationally established to collect and report data on streams during the timeframe covered by the evaluation. Only PTs maintained information on PNs at the level of program stream. Therefore, without consistent information from all PTs, the evaluation cannot present a comprehensive picture of PNs by stream. This limitation was anticipated in the evaluation design stage and a number of other sources were identified to provide some of the missing information. In particular, the International Region maintains detailed information on applications. Also, the PN survey offers some data on numbers by stream, as respondents were asked to identify the PNP stream under which they applied. However, the degree to which the stream breakdown of PN survey respondents accurately reflects the entire PN population is not known and it was not possible to weight the data by stream. As well, there is no way to validate the stream information as declared by PNs in the survey.

There are several data sources and periods covered for PN outcome information, which means that outcomes, such as incidence of employment and earnings, are reported using a variety of measures.

To address this limitation, IMDB was relied upon as the primary source of information for PN economic outcomes, as it is the most robust data available in that regard. Outcome information from IMDB is only available up to 2008, which is the most current tax year reported in the system. This time lag in this source was supplemented with the PN survey and other data, where applicable. Since IMDB data is not available by program stream, this information was also derived from the PN survey, where respondents were asked to identify the PNP stream under which they applied. Overall, the sources of information are clearly cited throughout the report so that the limitations can be taken into account, and corroborating evidence from multiple sources is also presented to strengthen the findings.

There may be biases in the PN survey.

The PN survey may be somewhat limited in its ability to assess outcomes since it was administered to those PNs who could be contacted at their most up-to-date address on file at CIC. This could bias the sample towards those who stayed in the PT of nomination, overestimating the retention rates^{Footnote 19}. Secondly, the PT survey might also be limited by the fact it was not accessible to those PNs who did not speak one of the seven languages in which the survey was administered. However, this is likely mitigated by the fact that the seven languages in which the survey was offered represents 64% of all PNs.

Employer and focus group respondents were not identified randomly.

This may introduce a potential bias towards those who had positive experiences with the PNP, making them more likely to have more favorable opinions to voice regarding the program. This limitation was anticipated at the design stage and, two mitigation strategies were applied: PT representatives were provided with criteria and guidelines for key informants and focus group participants, with standardized interview guides and protocols used to ensure consistency; and the evaluators were able to contact other key stakeholder organizations that they felt would offer additional perspectives.

• Previous
• 1
• 2
• 3
• 4
• 5
• 6 (current)
• 7
• 8
• 9
• 10
• 11
• Next