ARCHIVED – Enhanced Language Training Initiative: Formative Evaluation
The implementation of this formative evaluation leads to the identification of a number of issues for the summative evaluation planned for 2010 – most of which relate to the availability of data.
Measuring program rationale
Information on labour market challenges for newcomers is available from the 2003 Longitudinal Survey of Immigrants to Canada: A Portrait of Early Settlement Experiences. However, this longitudinal study is not expected to be repeated, so more up-to-date information on the issues addressed in the LSIC will not likely be available for the summative evaluation. Some data on immigrant employment is available from the 2001 census and will, as of December 2007, be available from the 2006 census. [ note 38] In addition, recently Statistics Canada released the first analysis of Canadian immigrants’ labour market performance based on the Labour Force Survey (LFS) [ note 39 ]. The report titled The Canadian Immigrant Labour Market in 2006: First Results from Canada’s Labour Force Survey provides key labour market indicators for immigrants in the labour market.
This will mean that the summative evaluation could rely on the 2006 census, the LFS, information from interviews and government documents to support the rationale for the program. However, this does not preclude the option of CIC partnering with other research initiatives (e.g., those carried out by Metropolis or the provinces/territories) to conduct specific research to support the assessment of the program’s rationale.
Describing participant profiles and measuring participant outcomes
As outlined in Section 3.5.4, ELT participant data – including participant profiles and outcomes – as it is currently being collected, was not adequate for this formative evaluation. Although more participant information should be available for the summative evaluation, unless the approach to collecting and inputting participant data is modified (e.g., the SPOs submitting all questionnaires at the end of the agreement, delays inputting data), there will likely still be missing participant questionnaires when the summative evaluation is underway. Also, once the records from the multi-year projects are available, there are remaining issues with the completeness and quality of the data provided.
In general, unless the issues of data comprehensiveness and quality are addressed, there will not be adequate program information for the summative evaluation. Without participant profile and outcome information, it will be difficult in the summative evaluation to identify successful models and, as is the case with this formative evaluation, the assessment will have to be based on qualitative information.
Determining cost-effectiveness and efficiency
Without adequate information on projects and participants, it will be difficult to assess the cost-effectiveness and efficiency of the program. In addition, the database, as it currently stands, does not provide information on project contributions that is easy to analyze, without considerable manual manipulations. One particular challenge with the existing cost data is that it does not distinguish between development and delivery costs. Even within delivery projects, SPOs have used ELT funding to develop curricula and tools for their delivery activities. It is important to be able to distinguish these two costs, in order to be able to identify the delivery costs, which can then be assessed in line with the number of participants and participant outcomes.
In addition, it will be necessary to identify a comparison program for assessing the cost-effectiveness of ELT. There are a number of programs that offer language training/employment assistance to newcomers (see Section 3.6). In the lead-up to the summative evaluation, it will be important to determine which programs are still being offered, whether they have components similar to ELT components and whether there is cost information available for these programs. If so, these programs may provide a comparison program for assessing cost-effectiveness for ELT (or components of ELT). If this information is not available, then cost-effectiveness may only be able to be assessed qualitatively by asking stakeholders if the identified costs are reasonable for the ELT outcomes. While this is not a strong methodology, it is still useful for the program to collect information on development and delivery costs to assess their reasonableness.
Identification and analysis of alternative programs of workforce integration
This formative evaluation attempted to collect information on alternative programs to support workforce integration for newcomers. The information that could be collected in one interview with a provincial representative was not adequate to describe, in any detail, other programs at the provincial level. At the federal level, the information was limited by the interviews that could be set up with representatives of other federal government departments. If the summative evaluation is to include a reasonably comprehensive review of alternative programs, this issue will have to be the subject of a specific data collection methodology that would include multiple interviews with federal and provincial/territorial representatives, as a well as the collection and review of program documents from these other jurisdictions. A fairly in-depth review is necessary to ensure the comparability of the programs, with respect to their objectives, approaches, eligibility requirements and outcomes. It would also need to be based on an agreed-upon set of criteria for assessing the comparability.
____________
[38] It includes information on employment status by period of immigration to Canada and is available with gender disaggregated.
[39] LFS is a monthly household survey used to produce estimates of labour market indicators such as unemployment, employment, and participation.
Page details
- Date modified: