Evaluation of the International Student Program

Appendix A: Evaluation framework – Logic model and evaluation matrix

Copy of the logic model (Appendix A) is available upon request to Research-Recherche@cic.gc.ca.

Evaluation matrix
Evaluation Issue	Indicators	Methodology
Relevance
Q1. Is the International Student Program aligned with CIC objectives and priorities with respect to migration?	Alignment with CIC’s legislative obligations Alignment of ISP to CIC objectives with respect to migration	Document review (IRPA, ISP and other documentation related to immigration mandate, RPP, DPR) Internal interviews^{Footnote 115} (Immigration, OMC, Strategic Policy and Priorities Branches)
Q2. Does the program support Government of Canada objectives and priorities, and is it consistent with related activities in other government departments?	Alignment of ISP to GoC objectives and priorities Alignment with related activities in other government departments Evidence of Canada’s economic need for migration strategies Perceptions of program partners (OGDs and P/Ts)	Document review (Throne Speech, Budget, Advantage Canada, documentation from OGDs) Internal interviews (Immigration, OMC, Strategic Policy and Priorities Branches) Partner interviews (DFAIT, CBSA, HRSDC, PCH, CIDA, IC, P/Ts)
Program results
Profile of the International Student Program
Q3. How many international students have come to Canada to study over time?	Number of international students arriving Trends over time	Existing databases (FOSS/CAIPS) Facts and Figures
Q4. Has their profile changed?	Profile of international students arriving (gender, level of study, length of study, country of origin, field of study, regional distribution by province and relation to MTV, urban and other) Profile of educational institutions receiving international students^{Footnote 116} Trends over time	Document review (ISP documentation and stakeholder/partner reports, including CBIE research, P/T websites and reports, DFAIT and OECD reports) Existing databases (FOSS/CAIPS) Facts and Figures Survey of international students (via CBIE)
Meeting Canada’s immigration objectives
Q5. Are Canada’s study and work opportunities for international students globally competitive?	Changes over time to Canada’s market share of international students Comparisons of Canada’s market share of international students to other comparable countries (taking into consideration such things as size of country and education sector) Changes over time in the number of international student arrivals Perceptions of international students and educational institutions International comparisons of policies, programs and operations (e.g. processing times, ease of application process, marketing and branding) related to international students^{Footnote 117}	Document review (CIC analysis and stakeholder/partner reports, including CBIE research, DFAIT branding work, IC report, NZ report) OECD statistics Administrative databases (FOSS/CAIPS) Survey of international students (via CBIE) Survey of educational institutions Literature review (documentation/websites on programs from comparable countries, such as UK, NZ, USA, Australia, France).
Q6. How successful has Canada been in attracting and retaining international students?	Changes over time to Canada’s market share of international students Changes over time in the number of international student arrivals Changes over time in the number of international students staying to work post graduation Number of international students applying for permanent residence (through various streams, including CEC, if feasible) Number of international students who become permanent residents Comparison of federal program with Quebec program (transitions from international student to PR by program and comparison of policies and programs)	Document review (ISP documentation, QC website and reports) Existing databases (FOSS/CAIPS) Facts and Figures OECD statistics Stakeholder reports (e.g. CBIE research) Survey of international students (via CBIE) to look at intentions to work post graduation and/or apply for permanent residence
Q7. Are international students taking advantage of work opportunities and gaining Canadian work experience?	Number of international students using work programs (OCWP, PGWP) Percentage of international students with a work permit who obtained employment Duration/frequency of work experience Quality of work experience (e.g. related to field of study, part-time during studies versus full-time post graduation)	Existing databases (FOSS) OMC statistics Stakeholder reports (e.g. CBIE research) Survey of international students (via CBIE)
Q8. What factors contribute to international students taking advantage of work opportunities and gaining Canadian work experience?	Differences in the profile of international students using the work programs and those not using them (e.g. gender, family wealth) Differences in the profile of international students obtaining employment and those not obtaining employment Perceptions and experiences of employers (including factors affecting their use of employment programs to hire international students) Perceptions and experiences of educational institutions (DIRs)	Existing databases (FOSS) Facts and Figures Stakeholder reports (e.g. CBIE research) Survey of international students (via CBIE) Survey of employers Survey of educational institutions
Q9. Is the profile of international students choosing to stay in Canada consistent with Canada’s immigration objectives?	Profile of international students in PGWP and applying for permanent residence (gender, level of study, regional distribution by province and relation to MTV, urban and other) Field of study of international students in PGWP and applying for permanent residence Number of international students in PGWP that apply and become permanent residents Comparison of federal program with Quebec program (transitions from international student to PR by program and comparison of policies and programs)	Document review (ISP documentation, QC website and reports) Existing databases (FOSS/CAIPS) OECD statistics Stakeholder reports (e.g. CBIE research) Survey of international students (via CBIE) to look at intentions to work post graduation and/or apply for permanent residence with link to field of study and other profile characteristics
Q10. What are the social, economic and cultural benefits of having international students studying and working in Canada?	Evidence of economic benefits to international students, educational institutions, communities and Canada (e.g. revenues to educational institutions, average amount of money spent by international students each year, number of international students who obtained employment, number of international students that become permanent residents, including regional distribution, trends over time, and their employment/economic earnings) Evidence of social and cultural benefits to international students, educational institutions, communities and Canada (e.g. diversity of international students, regional distribution of international students studying across Canada, participation of international students in cultural/social clubs/groups, perceptions of international students and educational institutions, distribution of international students across educational institutions, diversity and regional distribution of international students who become permanent residents)	Document review (reports/research from stakeholders/partners, including CBIE, AUCC, ACCC, educational institutions, P/Ts/QC, DFAIT/EduCanada) [Comment: Contact Pari Johnson at AUCC.] OECD statistics Administrative databases (FOSS) IMDB and other Statistics Canada data Survey of international students (via CBIE) Survey of educational institutions
Program integrity
Q11. Do program information and tools support quality decision-making?	Extent, timeliness and quality of functional guidance documentation, communications and field support Evidence of monitoring and data capture related to quality decision-making (including acceptance and refusal rates, reasons for refusals overseas, inland and POE) Effectiveness of existing mechanisms/procedures for quality decision-making (monitoring practices, data capture, strategic analysis, reports, bulletins/guidance to the field, information-sharing and coordination, etc.) Perceptions of CIC visa officers, CPC Vegreville officers and CBSA border services officers Evidence of inconsistent/inappropriate decision-making	Document review (OMC, IR, CPC Vegreville documentation, CIC website) Existing databases (FOSS/CAIPS) IR, OMC and Vegreville statistics/analyses Internal interviews (OMC Branch, IR, CPC Vegreville) Partner interview with CBSA Survey of CIC visa officers, CPC Vegreville officers and CBSA border services and program integrity officers Stakeholder interviews (ACISI, NEMR, F/P/T Groups) Survey of international students (via CBIE) Survey of educational institutions Survey of employers
Q12. Is there misuse and fraud in the International Student Program? What is being done to verify and deter program misuse and fraud? Are there alternatives to the current design and delivery of ISP that would improve program integrity?	Evidence of misuse and fraud by source (international students, educational institutions, third parties) Evidence of misuse and fraud by type of educational institution and level of study Number of cases referred to CBSA for investigation Evidence of monitoring and data capture related to misuse and fraud Effectiveness of existing mechanisms/procedures for fraud verification (monitoring practices, data capture, strategic analysis, reports, anti-fraud bulletins/guidance to the field, coordination and information-sharing within CIC and with CBSA, etc.) Success factors, gaps and barriers to effective verification of fraud Potential impacts of gaps and ambiguities in the legislation and regulations International comparisons of policies, programs and operations related to international students for the verification and prevention of misuse and fraud 3 Comparison of federal program with Quebec program (policies, programs and operations for the verification and prevention of misuse and fraud)	Document review (documentation on systems and procedures, reports by CIC and enforcement agencies, QC website and reports) Existing databases (CAIPS/FOSS) IR/OMC/Vegreville/CBSA statistics Internal interviews (Immigration and OMC Branches, IR, Regional Offices/RPAs and CPC Vegreville) Partner interviews (DFAIT, CBSA, RCMP and P/Ts/QC) Stakeholder interviews (ACISI, NEMR, F/P/T Groups) Survey of CIC visa officers, CPC Vegreville officers and CBSA border services and program integrity officers Literature review (documentation/websites on programs from comparable countries, such as UK, NZ, USA, Australia, France)
Program management
Q13. Is program delivery coordinated, timely and efficient?	Evidence of coordination and information-sharing within CIC and with CBSA (and/or other enforcement authorities) and P/Ts (OCWP) Extent, timeliness and quality of functional guidance documentation, communications and field support Changes in processing times / inventory (inland and overseas) Acceptance and refusal rates Level of stakeholder satisfaction with service Perceptions of CIC visa officers, CPC Vegreville officers and CBSA border services officers Success factors, gaps and barriers to coordinated, timely and efficient program delivery International comparisons of program delivery related to international students	Document review (OMC, IR, CPC Vegreville documentation, MOUs and other formal agreements) Existing databases (CAIPS/FOSS) IR/OMC/Vegreville/CBSA statistics Internal interviews (Immigration and OMC Branches, IR , CPC Vegreville, SIO) Partner interviews (DFAIT, CBSA and P/Ts) Stakeholder interviews (ACISI, NEMR, F/P/T Groups) Survey of CIC visa officers, CPC Vegreville officers and CBSA border services and program integrity officers Literature review (documentation/websites on programs from comparable countries, such as UK, NZ, USA, Australia, France)
Policy and program development
Q14. Is there a shared understanding of roles, responsibilities and ISP policy and program objectives across program partners and stakeholders?	Level of partner understanding and consensus with respect to roles, responsibilities and objectives Level of stakeholder (educational institutions, employers) understanding and consensus with respect to roles, responsibilities and objectives	Document review (meeting documentation from ACISI, NEMR, F/P/T Groups, MOUs and other formal agreements, other documentation from partners and stakeholders, including P/T and stakeholder reports and documents related to attracting and recruiting international students, as available) Internal interviews (Immigration Branch, OMC, IR, CPC Vegreville, Strategic Policy and Priorities, SIO) Partner interviews (DFAIT, CBSA, HRSDC, PCH, CIDA, IC and P/Ts) Stakeholder interviews (ACISI, NEMR, F/P/T Groups) Survey of educational institutions Survey of employers
Q15. Are program partners and stakeholders engaged and working together to advance the objectives of the International Student Program?	Evidence of meetings, consultations, presentations, working groups, MOUs and other formal agreements with partners and stakeholders Number/percentage of P/T jurisdictions partnering with CIC and of eligible educational institutions partnering with P/T jurisdictions (e.g. for the OCWP) Quality of mechanisms to engage program partners and stakeholders (e.g. ACISI, NEMR, F/P/T Groups), and formalize relationships (MoUs and other formal agreements with partners) Quality of relationships with partners and stakeholders Evidence of coordinated planning (commitment of necessary resources) Evidence of joint/collaborative frameworks and initiatives	Document review (ISP and other documentation related to immigration mandate, RPP, DPR , meeting documentation from ACISI, NEMR, F/P/T Groups, reports from stakeholders, MOUs and other formal agreements, reports from P/Ts) Internal interviews (Immigration Branch, OMC, IR, CPC Vegreville, Strategic Policy and Priorities) Partner interviews (DFAIT, CBSA, HRSDC, PCH, CIDA, IC and P/Ts) Stakeholder interviews (ACISI, NEMR, F/P/T Groups) Survey of educational institutions Survey of employers
Q16. Do ISP policies and programs address partner and stakeholder needs and facilitate study and work opportunities for international students?	Evidence of policy and program shifts/changes and their timeliness Evidence of measures planned/underway to improve policies and programs Evidence of policy issues not yet addressed or resolved Perceptions of international students, educational institutions and employers (including satisfaction/dissatisfaction with policies and programs) Number of international students arriving Number of international students using work programs	Document review (ISP and other documentation related to immigration mandate, meeting documentation from ACISI, NEMR, F/P/T Groups, reports from stakeholders, MOUs and other formal agreements, reports from P/Ts, new release database and daily rap) Existing databases (FOSS) Internal interviews (Immigration Branch, OMC, IR, CPC Vegreville, Strategic Policy and Priorities) Partner interviews (DFAIT, CBSA, HRSDC, PCH, CIDA, IC and P/Ts) Stakeholder interviews (ACISI, NEMR, F/P/T Groups) Survey of international students (via CBIE) Survey of educational institutions Survey of employers
Q17. Is the International Student Program based on a consistent, coherent and complementary policy and program framework?	Level of integration/coherence of ISP with other CIC objectives for migration Evidence of complementary policy objectives across other government departments and provinces/territories Evidence of competing priorities and inconsistent policies and programs within CIC and across other government departments and provinces/territories	Document review (ISP and other documentation related to immigration mandate, RPP, DPR , meeting documentation from ACISI, NEMR, F/P/T Groups, reports from stakeholders, MOUs and other formal agreements and reports from P/Ts) Internal interviews (Immigration Branch, OMC, IR, CPC Vegreville, Strategic Policy and Priorities) Partner interviews (DFAIT, CBSA, HRSDC, PCH, CIDA, IC, P/Ts and PCO) Stakeholder interviews (ACISI, NEMR, F/P/T Groups)
Cost-Effectiveness
Q18. Are costs in line with what would be expected in other similar programs?	Cost per undertaking ISP versus cost per undertaking Temporary Foreign Worker (TFW) Program^{Footnote 118} Review/comparison of costs of international approaches (e.g. Australia, UK, NZ, USA)^{Footnote 119}	Existing databases (CAIPS/FOSS) Cost Management Model Literature Review (documentation/websites on ISP and programs from comparable countries, such as UK, NZ, USA, Australia, France)
Q19. Are there alternatives to the current design and delivery of the International Student Program that would improve efficiency?	Review/comparison of alternative design/delivery approaches (e.g. Australia, UK, NZ, USA) Changes/improvements to ISP design and delivery Trends in processing times/inventory for international students Trends in the number of transactions with CIC (applications for study and work permits and for changes/renewal) Time allocated to ISP application processing by CIC visa officers relative to TFW application processing Relative time allocated to processing ISP applications for different types of educational institutions (university, college, private school)	Literature review (documentation/websites on ISP and programs from comparable countries, such as UK, NZ, USA, Australia, France) Internal Interviews (Immigration and OMC Branches, IR, CPC Vegreville, SIO) Survey of CIC visa officers, CPC Vegreville officers and CBSA border services and program integrity officers Existing databases (CAIPS/FOSS) OMC/IR/Vegreville statistics SIO reports (if available)
Q20. Have there been any unexpected outcomes?	Changes in policy and program direction Unintended results of programs and policies	Document review (audits, evaluations/reviews, reports) Internal interviews (Immigration and OMC Branches, IR and CPC Vegreville) Partner interviews (DFAIT, CBSA, HRSDC, PCH, CIDA, IC and P/Ts) Stakeholder interviews (ACISI, NEMR, F/P/T Groups)

Footnotes

Footnote 115

Internal interviews refer to interviews with CIC officials. Partner interviews refer to interviews with delivery partners including other government departments (CBSA and DFAIT principally, but also IC, HRSDC, PCH, CIDA) and provincial and territorial departments. Stakeholders refer to educational institutions, employers and international students and their representative associations.

Return to footnote115 referrer

Footnote 116

To the extent possible, a profile of educational institutions receiving international students will be explored through document review and developed in the evaluation.

Return to footnote116 referrer

Footnote 117

It is important to note that each of these countries has a national department of education, so they do not face the same jurisdictional issues as Canada.

Return to footnote117 referrer

Footnote 118