Evaluation of the Canadian Experience Class
2 Methodology
2.1 Evaluation Approach
The evaluation scope and approach were determined during a planning phase, in consultation with Citizenship and Immigration Canada (CIC) branches involved in the design, management and delivery of the Canadian Experience Class. This planning work included the development of a logic model and evaluation framework, including evaluation questions and indicators. Terms of Reference for the evaluation were approved by CIC’s Departmental Evaluation Committee in January 2014. The evaluation was conducted in-house, with a targeted contract for the administration of the survey of CEC PAs.
2.2 Evaluation Scope
The evaluation assessed the issues of relevance and performance of the Canadian Experience Class, and was guided by the program logic model, which outlines the expected immediate and intermediate outcomes for the program (see Appendix B: Logic model for the Canadian Experience Class. Evaluation questions were developed to address these core issues, and are presented in Table 2.1.
Table 2.1: Evaluation Questions
Relevance
- Is there a continued need for the CEC?
- Is the CEC aligned with CIC and government-wide priorities and objectives?
- Is the CEC consistent with federal roles and responsibilities?
Performance - Effectiveness
- To what extent are stakeholders and prospective immigrants informed about the CEC?
- To what extent are qualified international graduates and skilled temporary foreign workers applying for permanent residence under the CEC program?
- To what extent has the CEC increased Canada’s supply of skilled workers with Canadian education and / or work experience, and language proficiency?
- To what extent are CEC permanent residents staying in Canada and establishing economically?
- To what extent do CEC permanent residents integrate into the labour market quickly, and at a level commensurate with their skills?
Performance – Program Management and Resource Utilization
- To what extent does CEC management support effective program delivery?
- To what extent is the CEC process streamlined and timely? Has CIC’s modernization initiative had an impact?
- Are the program’s resources managed effectively to facilitate the achievement of outcomes?
- To what extent is CEC program delivery efficient?
Performance indicators were identified for each evaluation question to form the evaluation framework for the study. The evaluation issues, questions and performance indicators are presented in the Extended Evaluation Report.
2.3 Data Collection Methods
The evaluation questions and performance indicators were addressed using multiple lines of evidence to gather qualitative and quantitative data from a wide range of perspectives, including program managers, stakeholders and clients. These lines of evidence are presented in Table 2.2. More detailed information on the data collection methods used in the evaluation is provided in the Extended Evaluation Report.
Table 2.2: Lines of Evidence Used in the Evaluation
Line of Evidence
Literature and Document Review
The review examined relevant background policy and program documents, reports and research articles published in Canada and abroad on programs allowing for a transition to permanent residence, as well as the extent to which shortages of skilled labour exist in Canada. The document review provides background and context, informing the assessment of the relevance and performance of the CEC.
Interviews
Thirty (30) key informant interviews were conducted with CIC representatives (13) involved in the design and delivery of the program, as well as with members of other federal and provincial governments (7), and key external stakeholders (10), including representatives from employment-related and education-related organizations, and immigration representatives.
Administrative Data Analysis
Global Case Management System (GCMS), Computer-Assisted Immigration Processing System (CAIPS) and Field Operations Support System (FOSS) were used to develop the profile of applications and admissions under the CEC since the program inception up to 2014.
The Longitudinal Immigration Database (IMDB) was used to provide information on the economic establishment and interprovincial mobility of CEC PAs who were admitted between 2009 and 2012 and who filed a tax return within those years.Footnote 4
The Immigration Contribution Accountability Measurement System (iCAMS) and the Immigration Contribution Agreement Reporting Environment (iCARE) provided information on the use of settlement services by CEC PAs admitted between 2009 and 2013 who obtained services until the end of June 2014.Footnote 5
To provide information on CEC-related communications, summary reports of outreach activities by CIC region were reviewed, as well as sample products and some CIC website and call centre statistics.
In order to examine the resource utilization by the CEC, program delivery costs were examined in relation to the volume of applications processed. CIC Cost Management data, including federal processing costs for CEC immigrants, were used for this purpose.
Survey of CEC PAs
A survey was administered (by telephone and online) to 1,315 CEC PAs admitted to Canada between 2009 and 2013. The survey collected information on their experiences in applying to the program, their previous education and employment outside Canada and as a temporary resident, their employment and education as a permanent resident and their adaptation to life in Canada. The margin of error for the survey was ±2.6%, with a confidence interval of 95%.
Survey of Employers
A survey was administered online to 37 current or latest employers identified by CEC PAs in the survey of CEC PAs (described above).Footnote 6 The goal of this survey was to obtain the views of employers on their experience employing a CEC permanent resident, including their perception on how this person adapted to the work environment, and employer awareness of the CEC. The response rate for this survey was 20.1%.
Survey of Educational Institutions
A survey was administered online to 43 member institutions of the Association of Universities and Colleges of Canada (AUCC) and Colleges and Institutes Canada (C&IC) outside Quebec. The goal of this survey was to assess the degree of awareness of the CEC, as well as the extent of awareness-building efforts by educational institutions. The response rate for this survey was 24.6%.
2.4 Limitations and Considerations
Some limitations were noted in relation to the limited number of survey responses on the employer and educational institutions surveys, the representativeness of iCAMS and iCARE data, and the limited availability of data on long-term outcomes of CEC PAs. Various mitigation strategies were used to address the limitations and to ensure that the evaluation presented reliable information to support strong findings. These limitations and their corresponding mitigation strategies are described in more detail in the Extended Evaluation Report.
Overall, the evaluation design employed a balance of qualitative and quantitative methodologies, enabling the triangulation of research findings. The different lines of evidence were complementary and reduced information gaps, and generally, the various results converged towards common and integrated findings. The triangulation of the multiple lines of evidence, along with the mitigation strategies used in this evaluation, were considered sufficient to ensure that the findings are reliable and can be used with confidence.
Page details
- Date modified: