Evaluation of the Canada – Saskatchewan Labour Market Development Agreement

On this page

• List of abbreviations
• List of tables
• List of charts
• Executive summary
  • The LMDA investment
  • Evaluation objectives
  • Evaluation methodology
  • Key findings
  • Recommendations
• Management response
• Introduction
• Labour Market Development Agreement
  • Employment benefits
  • Support measures
  • Eligible participants covered in this study
• Methodology
  • Incremental impacts analysis
  • Factors accounted for in the cost-benefit analysis
  • Strengths and limitations of the studies
  • Overview of the studies summarized in this report
• Evaluation findings
  • Profile of participants
  • Incremental impacts for active and former EI claimants
  • Incremental impacts by subgroups of participants
  • Cost-benefit analysis
• Supplemental studies
  • Self-Employment
  • Skills Training-Apprentices
• Conclusions and recommendations
  • Recommendations
• References
• Appendix A. List of 5 studies included in this synthesis report

Alternate formats

Evaluation of the Canada – Saskatchewan Labour Market Development Agreement [PDF - 1,02 KB]

Large print, braille, MP3 (audio), e-text and DAISY formats are available on demand by ordering online or calling 1 800 O-Canada (1-800-622-6232). If you use a teletypewriter (TTY), call 1-800-926-9105.

• List of abbreviations

  EBSM

  Employment Benefits and Support Measures

  EI

  Employment Insurance

  ESDC

  Employment and Social Development Canada

  LMDA

  Labour Market Development Agreements

  SA

  Social assistance

• List of tables

  • Table i. Share of LMDA funding and average cost per Action Plan Equivalent per participant in Saskatchewan, for 2010 to 2012 period
  • Table ii. Number of years for the benefits to exceed program costs
  • Table 1. Share of LMDA funding and average cost per Action Plan Equivalent in Saskatchewan
  • Table 2. Profile of active and former EI claimant participants in EBSMs in 2010 to 2012 in Saskatchewan
  • Table 3. Employment and earning levels, and use of SA in the year before participation in EBSMs
  • Table 4. Profile of female active and former EI claimant participants in EBSMs in Saskatchewan in 2010 to 2012
  • Table 5. Incremental impacts for female participants
  • Table 6. Profile of male active and former EI claimant participants in EBSMs in Saskatchewan in 2010 to 2012
  • Table 7. Incremental impacts for male participants
  • Table 8. Profile of youth active and former EI claimant participants in EBSMs in Saskatchewan in 2010 to 2012
  • Table 9. Incremental impacts for youth participants
  • Table 10. Profile of older worker active EI claimant participants in EBSMs in Saskatchewan in 2010 to 2012
  • Table 11. Incremental impacts for older worker participants in Workforce Development
  • Table 12. Profile of Indigenous active and former EI claimant participants in EBSMs in Saskatchewan in 2010 to 2012
  • Table 13. Incremental impacts for Indigenous participants
  • Table 14. Incremental impacts for Skills Training participants in Regina/Saskatoon
  • Table 15. Incremental impacts for Skills Training participants outside of Regina/Saskatoon
  • Table 16. Cost-benefit results for active and former EI claimant participants
  • Table 17. Profile of active and former EI claimant participants in Skills Training-Apprentices in Saskatchewan in 2010 to 2012

• List of charts

  • Chart i. Change in probability of being employed in participants relative to non-participants (annual average)
  • Chart ii. Employment earnings of participants relative to non-participants (annual average)
  • Chart iii. Change in dependence on government income support (annual average)
  • Chart 1. Change in probability of being employed in participants relative to non-participants (annual average)
  • Chart 2. Employment earnings of participants relative to non-participants (annual average)
  • Chart 3. Change in the use of EI benefits (annual average)
  • Chart 4. Change in the use of SA benefits (annual average)
  • Chart 5. Change in dependence on government income support (annual average)
  • Chart 6. Average earnings for active claimant participants in Skills Training-Apprentices
  • Chart 7. Average earnings for former claimant participants in Skills Training-Apprentices

Executive summary

The Canada-Saskatchewan Labour Market Development Agreement (LMDA) is a bilateral agreement between Canada and Saskatchewan for the design and delivery of Employment Benefits and Support Measures (EBSMs).

The objective of EBSMs is to assist individuals to obtain or keep employment through various active employment programs, including training or employment assistance services. Successful delivery of EBSMs is expected to result in participants receiving needed services, a quick return to work, and savings to the Employment Insurance (EI) account.

Programs and services delivered by Saskatchewan have to correspond to the EBSM categories defined under the EI Act. The following is a short description of the EBSMs examined in the evaluation:

• Skills Training helps participants obtain employment skills by giving them financial assistance to attend classroom training
• Self-Employment provides financial assistance and business planning advice to participants to help them start their own business
• Workforce Development supports individuals as they prepare to enter or re-enter the workforce or assist them to find a better job
  • services can include counselling, job search skills, job placement services, provision of labour market information and case management

The LMDA investment

In fiscal year 2020 to 2021, Canada transferred approximately $54 million (including nearly $6 million in administration funds) to Saskatchewan.

Table i provides an overview of the share of funding allocated to EBSMs and the average cost per Action Plan Equivalent for active and former EI claimants. The average cost per participant is calculated based on the 2010 to 2012 data from the EI Monitoring and Assessment Reports. The 2010 to 2012 period corresponds with the cohort of participants selected for incremental impacts and cost-benefit analysis in the LMDA evaluation.

Table i. Share of LMDA funding and average cost per Action Plan Equivalent per participant in Saskatchewan, for 2010 to 2012 period^{Footnote 1},^{Footnote 2}

Employment Benefits and Support Measures	Average share of funding	Average cost – active claimants	Average cost – former claimants
Skills Training	73%	$5,128	$5,423
Workforce Development	15%	$708	$708
Labour Market Partnerships	6%	n/a	n/a
Targeted Wage Subsidies	3%	$5,438	$5,160
Self-Employment	1%	$5,098	$4,995
Research and Innovation	1%	n/a	n/a
Total	99%	n/a	n/a

Sources: EI Monitoring and Assessment Reports for fiscal years 2010 to 2011 and 2011 to 2012.

Note: total spending may not add up to 100% due to rounding.

Compared to the 2010 to 2012 period, the LMDA budget allocation varied for few programs and services in 2020 to 2021. For example, investments in Skills Training decreased from 73% to 61%. As well, investments in Workforce Development increased from 15% to 24% of total allocation.

Evaluation objectives

Building on the success of previous LMDA evaluation cycles, the aim of this evaluation is to fill in knowledge gaps about the effectiveness, efficiency as well as design and delivery of EBSMs in Saskatchewan.

Evaluation methodology

The findings in this report are drawn from 5 separate evaluation studies carried out at the provincial level. These studies examine issues related to program effectiveness, efficiency, and design and delivery. A mix of qualitative and quantitative methods are used, including:

• incremental impact analysis for participants who began an intervention between 2010 and 2012
• outcome analysis
• cost-benefit analysis (including savings to health care)
• 6 key informant interviews with 7 program managers and service providers
• provincial questionnaires
• a survey of 49 Self-Employment participants
• document and literature reviews

The incremental impacts are estimated for 2 types of EI claimants:

• active claimants are participants who started an EBSM intervention while collecting EI benefits
• former claimants are participants who started an EBSM intervention up to 3 years after the end of their EI benefits^{Footnote 3}

Key findings

In Saskatchewan, nearly 13,800 EI active and former claimants began participating in LMDA programs and services between 2010 and 2012.

Effectiveness and efficiency of EBSMs

Overall, incremental impacts demonstrate that participation in most EBSMs improves labour market attachment and reduces dependence on government income supports compared to similar non-participants. These results are consistent with those found for earlier cohorts of participants as part of the previous evaluation cycle. A subgroup analyses shows that with some exceptions, Skills Training improves the labour market attachment and reduced the dependence on income support for most subgroups of participants. Workforce Development alone was found to improve the labour market attachment for female, male, youth, older workers, and Indigenous participants. As well, the social benefits of participating in EBSMs exceeds the initial investment costs over time.

Chart i presents the incremental impacts on the incidence of employment for active and former claimants by EBSM. The estimates can be interpreted as a change in the probability of being employed following participation. For example, participation in Skills Training increases the probability of being employed by 8.8 percentage points for active EI claimants relative to non-participants.

Note: Impacts are estimated over 4 post-program years (or 5 years in the case of Workforce Development).

Text description – Chart i

Program name	Active claimants (percentage points)	Former claimants (percentage points)
Skills Training	8.8	5.9
Workforce Development	4.2	n/a

Note: Impacts are estimated over 4 post-program years (or 5 years in the case of Workforce Development).

Chart ii presents the annual average increase in employment earnings for active and former claimants over the post-participation period.

Note: Impacts are estimated over 4 post-program years (or 5 years in the case of Workforce Development).

Text description – Chart ii

Program name	Active claimants	Former claimants
Skills Training	$7,760	$3,698
Workforce Development	$2,356	n/a

Note: Impacts are estimated over 4 post-program years (or 5 years in the case of Workforce Development).

As shown in Chart iii, overall active and former claimants reduce their dependence on government income supports.

*The impact is not statistically significant over the entire post-program period.

Note: Impacts are estimated over 4 post-program years (or 5 years in the case of Workforce Development).

Text description – Chart iii

Program name	Active claimants (percentage points)	Former claimants (percentage points)
Skills Training	-7.2	-5.3
Workforce Development	-0.9*	n/a

*The impact is not statistically significant over the entire post-program period.

Note: Impacts are estimated over 4 post-program years (or 5 years in the case of Workforce Development).

Table ii presents the number of years required for the social benefits to exceed program costs. Social benefits to participation exceed initial investment costs over a period ranging from 1 year to 1.4 years.

Table ii. Number of years for the benefits to exceed program costs

Category	Skills Training active claimants (10 years post-program)	Workforce Development active claimants (5 years post-program)	Skills Training former claimants (10 years post-program)
Payback period (years after end of participation)	1.2	1.4	1.0

Supplemental studies

Self-Employment study

A supplemental study addresses information gaps previously identified about the Self-Employment program. The study addressed program design and delivery, challenges and lessons learned.

The evaluation found that the Self-Employment program aims to assist participants in creating employment for themselves by providing them with a range of services.

Based on a survey, it was found that 2 to 4 years after program participation:

• nearly 68% of survey respondents who launched a self-employment business were still in operation in winter 2020
• 70% of respondents said that they are financially about the same or better off after the program
• 68% of respondents said that their household net worth is about the same or higher after the program

The survey did examine the contribution of the program to the success of self-employment businesses. At least 95% of survey respondents who launched a self-employment business rated the following services and training as very or somewhat important to the business launch, operation, and success:

• orientation or information sessions
• assessment of entrepreneurial suitability
• financial assistance for business start-up costs
• one-on-one mentorship
• living allowance during participation

Skills Training-Apprentices study

The objective of the program is to help apprentices become skilled tradespeople and to increase their labour market attachment. Program participants have generally chosen a career and are already attached to the labour market. The apprenticeship process involves on-the-job learning and technical training in a classroom setting.

The evaluation found that active EI claimants increased their average earnings from $18,590 in the fifth year pre-program to $64,293 in the fifth year after the program start year. Former EI claimants increased their average earnings from $20,900 in the fifth year pre-program to $58,137 in the fifth year after the program start year. After participating in the program, both active and former claimants also decreased their dependence on government income supports.

Recommendations

Since 2012, 15 qualitative and quantitative studies addressed issues and questions related to EBSM design, delivery, and effectiveness across Canada. The quantitative studies successfully assessed the effectiveness and efficiency of EBSMs by producing incremental impacts and cost-benefit analysis. The qualitative studies help to contextualize the findings from the quantitative studies and to identify specific challenges, lessons learned, and best practices associated with the design and delivery of EBSMs. Each study included key considerations for program and policy development or recommendations.

In addition, the recently completed evaluation of the Workforce Development Agreements complements the LMDA qualitative studies. This comprehensive evaluation provided unique insights into challenges and lessons learned to assist persons with disabilities, immigrants and those further removed from the labour market.

Most results from this evaluation stem from the conduct of advance causal analysis whereby impacts found could be attributed to a specific EBSM. These analyses are predicated on having access to high quality administrative records, thereby confirming the importance of the capacity to leverage and integrate relevant administrative data.

From these main findings, 2 key recommendations emerge for Saskatchewan:

Recommendation #1: ESDC and Saskatchewan are encouraged to share and discuss lessons learned, best practices and challenges associated with the design and delivery of EI-funded programming. Discussions are encouraged at the bilateral or multilateral levels as well as with service delivery network if necessary.

Recommendation #2: ESDC and Saskatchewan are encouraged to pursue efforts to maintain and strengthen data collection provisions in support of reporting, performance measurement and data-driven evaluations at the national and provincial levels.

Management response

Employment and Social Development Canada collaborated with Saskatchewan, and other provinces and territories, during the planning and implementation of the third cycle of the LMDA evaluation. Saskatchewan agrees with the recommendations articulated in the report. Key actions for addressing the recommendations have been outlined in the management response below.

Recommendation #1: ESDC and Saskatchewan are encouraged to share and discuss lessons learned, best practices and challenges associated with the design and delivery of EI-funded programming. Discussions are encouraged at the bilateral or multilateral levels as well as with service delivery network if necessary.

Response: Agree

• Findings in the evaluation report demonstrate that LMDA programming delivered by Saskatchewan leads to positive labour market outcomes as former program participants report improved labour market attachment and reduced dependence on government income supports compared to similar non-participants
• Saskatchewan will continue collaborating with ESDC and other provinces and territories to arrange bilateral and/or multilateral discussions on the best practices and lessons learned on the delivery of EI-funded programs
• Saskatchewan will continue discussions with community-based organizations and network delivery stakeholders to collect and analyse insights on best practices and lessons learned

Recommendation #2: ESDC and Saskatchewan are encouraged to pursue efforts to maintain and strengthen data collection provisions in support of reporting, performance measurement and data-driven evaluations at the national and provincial levels.

Response: Agree

• Saskatchewan will continue working with ESDC on maintaining and strengthening data collection in support of reporting and performance measurement at the national and provincial levels. This also includes improvements of two-way data-sharing
• Saskatchewan will continue working with service delivery partners to ensure they are well equipped to collect, compile, and share data necessary to assess the results of LMDA programming

Introduction

Employment and Social Development Canada (ESDC) worked jointly with Saskatchewan and 11 other provinces and territories to undertake the 2018 to 2023 third cycle for the Labour Market Development Agreement (LMDA) evaluations.

The first cycle of LMDA evaluations was carried out from 1998 to 2012. It involved the conduct of separate formative and summative evaluations in all provinces and territories under the guidance of bilateral Joint Evaluation Committees.

Building on lessons learned and best practices from the first cycle, the second cycle of LMDA evaluations was undertaken between 2012 and 2017. The second cycle was designed and implemented under the guidance of a federal-provincial/territorial LMDA Evaluation Steering Committee. The work was supported by bilateral discussions at Joint Evaluation Committees.

The third LMDA evaluation cycle builds on the success of the second cycle. The aim is to fill in knowledge gaps about the effectiveness, efficiency, and design and delivery of Employment Benefits and Support Measures (EBSMs). The evaluation cycle was designed and implemented under the guidance of a federal-provincial/territorial LMDA Evaluation Steering Committee composed of ESDC and 12 participating provinces and territories.

For Saskatchewan, this report presents a summary of the third cycle evaluation findings from 5 studies.

Labour Market Development Agreements

The LMDAs are bilateral agreements between Canada and each province and territory for the design and delivery of EBSM programs and services. They were established under Part II of the 1996 Employment Insurance (EI) Act.

In fiscal year 2020 to 2021, Canada transferred nearly $54.4 million to Saskatchewan. Under the Canada-Saskatchewan LMDA, Saskatchewan is responsible for the design and delivery of programs and services aimed at assisting individuals to prepare for, obtain, and maintain employment.

LMDA programs and services are classified under 2 categories:

• Employment benefits^{Footnote 4}, ^{Footnote 5} fall into 5 sub-categories: Skills Training, Targeted Wage Subsidies, Self-Employment, Job Creation Partnerships, and Targeted Earnings Supplements ^{Footnote 6}
• Support measures fall into 3 sub-categories: Workforce Development^{Footnote 7}, Labour Market Partnerships, and Research and Innovation

Saskatchewan has the flexibility to adapt EBSMs to their jurisdiction context as long as they are consistent with Part II of the EI Act.^{Footnote 8}

The objective of EBSMs is to assist individuals to obtain or keep employment through various active employment programs, including training or employment assistance services. Successful delivery of EBSMs is expected to result in participants receiving needed services, a quick return to work, and savings to the EI account.

Employment benefits

The following employment benefits program is examined in this study:

Skills Training provides direct financial assistance to individuals to select, arrange, and pay for training. Training is tailored to the needs of participants through counselling and career orientation. It can include adult-based education, literacy and essential skills, language training, short-term training and occupational training leading to certification from an accredited institution.

Support measures

The following support measures program is examined in this study:

Workforce Development supports individuals as they prepare to enter or re-enter the workforce or assist them to find a better job. Services include job search services, career development and counselling, and résumé writing assistance. These services are light touch interventions due to their very short duration and can be provided on a one-on-one basis or in a group setting. A typical intervention lasts less than 1 day, but a participant may receive multiple short interventions over a few weeks. These services are generally provided in combination with more intensive interventions.

Eligible participants covered in this study

The incremental impacts are estimated for active and former EI claimants:

• active claimants are participants who started an EBSM intervention while collecting EI benefits
• former claimants are participants who started an EBSM intervention up to 3 years after the end of their EI benefits^{Footnote 9}

Average EBSM share of funding and cost per Action Plan Equivalent

Table 1 provides an overview of the share of funding allocated to EBSMs and the average cost per Action Plan Equivalent for active and former claimants. It is noted that the average cost per participant is calculated based on the 2010 to 2012 data from the EI Monitoring and Assessment Reports. The 2010 to 2012 period corresponds with the cohort of participants selected for incremental impacts and cost-benefit analysis.

From the 2010 to 2012 time period to the 2020 to 2021 fiscal year investments in Skills Training decreased by 12 percentage points. Investments in Workforce Development and Labour Market Partnerships increased by 9 percentage points each.

Table 1. Share of LMDA funding and average cost per Action Plan Equivalent in Saskatchewan^{Footnote 10},^{Footnote 11}

Employment Benefits and Support Measures	Share of funding (2010 to 2012)	Share of funding (2020 to 2021)	Average cost – active claimants (2010 to 2012)	Average cost – former claimants (2010 to 2012)
Skills Training	73%	61%	$5,128	$5,423
Workforce Development	15%	24%	$708	$708
Labour Market Partnerships	6%	15%	n/a	n/a
Targeted Wage Subsidies	3%	n/a	$5,438	$5,160
Self-Employment	1%	N/A	$5,098	$4,995
Research and Innovation	1%	❮1%	n/a	n/a

Sources: EI Monitoring and Assessment Reports for fiscal years 2010 to 2012 and 2020 to 2021.

Methodology

This section presents key aspects of the quantitative analyses carried out as part of the LMDA studies.

All quantitative analyses are based on administrative data from the EI Part I (EI claim data) and Part II (EBSM participation data). The EI Part I and II data are then linked to the T1 and T4 taxation files from the Canada Revenue Agency. Incremental impact and cost-benefit analyses are based on up to 100% of participants in Saskatchewan who began their EBSM participation in 2010 to 2012.

The 2010 to 2012 timeframe was selected to assess the impacts of EBSMs in the years following participation. Impacts were assessed over a period of at least 4 years after program completion up to the 2017 calendar year (most recent available information at the time of this evaluation).

Incremental impacts analysis^{Footnote 12}

Program effectiveness is assessed by estimating incremental impacts from EBSM participation on participants’ labour market experience. That is, earnings from employment and self-employment, incidence of employment, use of EI, use of social assistance (SA), and dependence on government income supports after participation.

The role of the incremental impact analysis is to isolate the effects of participation from other factors. To achieve this, the incremental impact analysis compares the labour market experience of participants before and after their participation with that of similar non-participants. Figure 1 presents an example of incremental impact calculation.

Text description – Figure 1

Incremental impacts are the difference in the pre-post participation outcomes between participants and similar non-participants.

For example:

• if the average annual earnings of participants are $30,000 before participation and $38,000 after participation, the change in earning for participants is an increase of $8,000
• if the average annual earnings in the comparison group are $31,000 before participation and $36,000 after participation, the change in earnings for the comparison group is an increase of $5,000
• the incremental impact, that is the change attributed to program participation, is $8,000 minus $5,000, which equals an increase of $3,000

The main estimator used is propensity score kernel matching technique combined with difference-in-differences estimator. Moreover, 3 different state-of-the-art estimation techniques (Inverse Probability Weighting, Nearest Neighbour and Cross-sectional Matching) were carried out separately for each type of EBSMs and EI claimants to validate the impact estimates.

As for previous LMDA evaluation studies, the Action Plan Equivalent is the unit of analysis used. Action Plan Equivalents regroup all EBSMs received by an individual within less than 6 months between the end of one EBSM and the start of the next. Action Plan Equivalents are categorized based on the longest EBSM they contain, except for Workforce Development-only Action Plan Equivalents which include only Workforce Development interventions.

The matching of participants and comparison group members used up to 75 socio-demographic and labour market variables observed over 5 years before participation. Two different comparison groups were used to measure impacts for active and former EI claimants.

For active claimants, incremental impacts were measured relative to a comparison group of active claimants who were eligible to, but did not, participate in EBSMs during the reference period.

For former claimants, the comparison group was created using individuals who participated in Workforce Development only during the reference period.^{Footnote 13} In other words, the experience of former claimants in Skills Training interventions is compared to the experience of former claimants who received Workforce Development only. This is a conservative approach given the fact that participation in Workforce Development can lead to limited effects on labour market outcomes.

Due to this difference in measurement, incremental impacts estimated for active claimant participants should not be directly compared to those of former claimant participants.

Impacts are generated over 4 years for Skills Training, while a fifth year is estimated for participants in Workforce Development.^{Footnote 14}

Factors accounted for in the cost-benefit analysis^{Footnote 15},^{Footnote 16}

Building on the results of the incremental impacts, program efficiency is assessed through a cost-benefit analysis. The analysis compares the participants’ cost of participating and the government’s cost of delivering the program to the benefits associated with the program. Overall, this analysis provides insights on the extent to which the program is efficient for the society (that is, for both participants and the government).

Sources of data and information

The analysis takes into account all the quantifiable costs and benefits directly related to EBSM delivery and participation that can be measured given the information available. The analysis is comprehensive in that it accounts for the vast majority of possible direct costs and benefits. However, the analysis does not account for all costs and benefits. For example, there are factors that can lead to an understatement of the benefits (for example, positive spillovers to other family members) and other factors that can lead to an overstatement of the benefits (for example, effects on skill prices or displacement).

This study relied on integrated data from the EI Part I and II Databank and Income Tax records from the Canada Revenue Agency. Information about earnings, use of EI, and use of SA was taken from the study of incremental impacts.^{Footnote 17} The program costs were calculated using information available in the EI Monitoring and Assessment Reports.

Relative to the previous cycle of evaluation, the methodology has been extended to incorporate one of the indirect health benefits associated with increased labour market attachment. In particular, the methodology includes an estimate of the change in public health care cost due to the decline in health care utilization resulting from program participation.

Data on average public healthcare costs by income quintiles are taken from the report Lifetime Distributional Effects of Publicly Financed Health Care in Canada (2013) by the Canadian Institute for Health Information.

Program costs are measured using information on LMDA expenditures and new interventions reported in the EI Monitoring and Assessment Report. Other costs and benefits are assessed using integrated administrative data from the EI Part I and II databank and the Canada Revenue Agency. Incremental impacts measured over the second year of participation and up to 5 post-program years are discounted by 3% to bring them to a common base with the program cost and benefits incurred in the program start year. This 3% rate accounts for the interest the government could have collected if the funds used to pay for the program had been invested. Incremental impacts are estimated using 2010 constant dollars and this accounts for inflation.

The costs and benefits accounted for in the calculations

Program cost: cost incurred by the government for delivering the program (that is, administration and direct program costs calculated from data reported in the EI Monitoring and Assessment Reports).

Marginal social cost of public funds: loss incurred by society when raising additional revenues such as taxes to fund government spending. The value is estimated as 20% of the program cost, sales taxes, income taxes, impacts on EI and impacts on SA paid or collected by the government.

Foregone earnings: estimated net impacts on participants’ earnings during the participation period. During labour market program participation, some individuals have lower earnings than what they would have received if they had not participated.

Employment earnings: incremental impacts on participants’ earnings during and after participation. In-program earnings represent the foregone earnings for participants.

Fringe benefits: the employer-paid health and life insurance as well as pension contributions. They are estimated at 15% of the incremental impacts on earnings.

Federal and provincial income taxes: incremental impacts on federal, provincial, and territorial taxes paid by participants.

Sales taxes: the sales taxes paid by participants estimated as incremental impacts on earnings multiplied by the propensity to consume (97%), the proportion of household spending on taxable goods and services (52%) and the total average federal and provincial sales tax rate (11%).

Social assistance and Employment Insurance benefits collected: incremental impacts on SA and EI benefits use by participants following participation.

Canada Pension Plan contribution and Employment Insurance premiums: these contributions and premiums were identified from the Canada Revenue Agency data and then, the incremental impacts on Canada Pension Plan contributions and EI premiums were estimated.

Public health care costs savings: estimated impact of participation in EBSMs on public health care costs shown as an average savings per participant over the post-program period examined.

Strengths and limitations of the studies

One of the key strengths from the studies is that all quantitative analyses are based on administrative data rather than survey responses. Compared to survey data, administrative data are not subject to recall errors or response bias.

The propensity score models used to match participants and non-participants for the incremental impact analyses are judged to be robust. In part this is because they were based on 5 years of pre-participation data. Moreover, these models are based on a vast array of variables including sociodemographic characteristics, location, skill level related to last occupation, and indicators of labour market attachment.

However, the matching process can be further refined for specific subgroups if the following information was available in the future, broadening the scope for greater Gender-based Analysis Plus examination:

• persons with disabilities: the type and severity of the disability, and the capacity/willingness to work full-time
• recent immigrants: the country of origin, the proficiency in English or French, and the relevance of credentials and work experience
• visible minorities: place of birth; individuals who are born outside of Canada face different challenges compared to those born in Canada

Refining the matching process for population subgroups could broaden the scope for greater Gender-based Plus Analysis.

Sensitivity analysis and the use of alternative estimation methods have increased confidence in the incremental impact estimates. However, one limitation with the propensity score matching techniques is that no one can be fully sure the impacts are not influenced by factors not captured in the data.

The cost-benefit analysis accounted for all quantifiable costs and benefits directly attributable to the EBSMs and could be estimated with the available administrative data. It is furthered strengthened by incorporating one of the indirect benefits, which is the change in public health care associated with program participation. However, the analysis did not account for non-quantifiable factors that can lead to an understatement of the benefits (for example, positive spillovers to other family members) and factors that can lead to an overstatement of the benefits (for example, effects on skill prices or displacement).

In some studies that use qualitative data collection methods, the number of key informants interviewed is relatively small in some Saskatchewan. Responses provided by key informants reflect their own experience and may not be fully representative of Saskatchewan.

Overview of the studies summarised in this report

The findings in this report are drawn from 5 separate studies carried out at the provincial level. These studies examine issues related to program effectiveness, efficiency, design/delivery and used a mix of qualitative and quantitative methods. Appendix A presents an overview of these studies. The studies are:

• Examination of the medium-term outcomes from 2010 to 2017
• Estimation of the medium-term incremental impacts from 2010 to 2017
• Cost-benefit analysis of Employment Benefits and Support Measures in Saskatchewan
• Cost-Benefit Analysis: Incorporating Public Health Care Costs Savings in the Context of the Labour Market Programs Evaluation in Saskatchewan
• Design and delivery of the Self-Employment program in Saskatchewan

Evaluation findings

Profile of participants

In Saskatchewan, nearly 13,800 EI active and former claimants participated in LMDA programs and services between 2010 and 2012.

The profile of participants is presented in Table 2 by gender, age, sociodemographic group, and marital status. Information about their educational attainment, occupation and industry is based on the latest job they held prior to applying for EI benefits. Information about sociodemographic groups is self-reported.

Table 2. Profile of active and former EI claimant participants in EBSMs in 2010 to 2012 in Saskatchewan

Categories	Active claimants	Former claimants
Number of participants	6,747	6,978
Gender: Female	43%	46%
Gender: Male	57%	54%
Age: 30 and under	34%	36%
Age: 31 to 54	57%	57%
Age: 55 and over	9%	7%
Sociodemographic group: Indigenous people	24%	37%
Sociodemographic group: Persons with disabilities	12%	15%
Sociodemographic group: Visible minorities	9%	10%
Sociodemographic group: Recent immigrants	4%	2%
Marital status: Single	48%	56%
Marital status: Married or common-law	36%	26%
Marital status: Widow / divorced / separated	12%	12%
Education or skills level: High school or occupational training	38%	36%
Education or skills level: College, vocational education, or apprenticeship training	26%	25%
Education or skills level: On-the-job training	25%	32%
Education or skills level: University degree	5%	4%

Note: Values may not equal 100% due to rounding or missing information.

Top 3 occupational groups for participants in EBSMs

Active claimants:

• semi-skilled manual workers = 18%
• other manual workers = 16%
• intermediate sales and service personnel = 11%

Former claimants:

• other manual workers = 17%
• other sales and service personnel = 15%
• intermediate sales and service personnel = 14%

Top 3 industries for participants in EBSMs^{Footnote 18}

Active claimants:

• construction = 15%
• public administration = 11%
• retail trade = 10%

Former claimants:

• construction = 13%
• accommodation and food services; and retail trade = 12% each
• public administration = 10%

As presented in Table 3, in the year before program participation, former claimants have lower levels of employment and earnings than active claimants. Former claimants also have a higher dependence on SA.

Table 3. Employment and earning levels, and use of SA in the year before participation in EBSMs

Pre-EBSM participation employment characteristics	Active claimants	Former claimants
Average employment earnings	$26,719	$13,647
Percentage employed	99%	83%
Percentage on SA	13%	30%

Incremental impacts for active and former EI claimants

The incremental impact results presented below are generally consistent with those found as part of the second LMDA evaluation cycle.

Incidence of employment

Chart 1 presents the incremental impacts on the incidence of employment^{Footnote 19} for active and former claimants by type of program. The estimates can be interpreted as a change in the probability of being employed following participation.

Active claimants in Skills Training and Workforce Development increased their incidence of employment relative to similar non-participants. Former claimants in Skills Training increased their incidence of employment relative to similar participants who received only Workforce Development services.

Note: Impacts are estimated over 4 post-program years (or 5 years in the case of Workforce Development).

Text description – Chart 1

Program name	Active claimants (percentage points)	Former claimants (percentage points)
Skills Training	8.8	5.9
Workforce Development	4.2	n/a

Note: Impacts are estimated over 4 post-program years (or 5 years in the case of Workforce Development).

Employment earnings

Chart 2 presents the average annual increase in employment earnings for active and former EI claimants over the 4 years post-participation.

Active EI claimants in Skills Training and Workforce Development increased their employment earnings compared to similar non-participants.

Former EI claimants in Skills Training increased their employment earnings relative to similar participants who received only Workforce Development services.

Note: Impacts are estimated over 4 post-program years (or 5 years in the case of Workforce Development).

Text description – Chart 2

Program name	Active claimants	Former claimants
Skills Training	$7,760	$3,698
Workforce Development	$2,356	n/a

Note: Impacts are estimated over 4 post-program years (or 5 years in the case of Workforce Development).

Use of EI benefits

As shown in Chart 3, active claimants in Workforce Development reduced their use of EI benefits in the post-program period compared to similar non-participants. Active claimants in Skills Trainings reduced their use of EI benefits however the impacts are not statistically significant. In the post-program period, former claimants in Skills Training, increased their EI benefits use relative to similar participants who received Workforce Development services only.

From a cost-benefit perspective, the increase in the use of EI by former claimants in Skills Training is consistent with previous evaluations and is not necessarily a negative impact given the increase in earnings. Following participation, former claimants are likely to requalify for EI benefits due to their stronger labour market attachment demonstrated by increases in their incidence of employment and earnings.

*The impact is not statistically significant over the entire post-participation period.

Note: Impacts are estimated over 4 post-program years (or 5 years in the case of Workforce Development).

Text description – Chart 3

Program name	Active claimants	Former claimants
Skills Training	-$157*	$350
Workforce Development	-$210	n/a

*The impact is not statistically significant over the entire post-participation period.

Note: Impacts are estimated over 4 post-program years (or 5 years in the case of Workforce Development).

Use of SA benefits

As shown in Chart 4, active and former EI claimants in Skills Training decreased their use of SA benefits in the post-program period.

Active claimants in Workforce Development increased their use of SA benefits slightly, however the impact is not statistically significant.

*The impact is not statistically significant over the entire post-participation period.

Note: Impacts are estimated over 4 post-program years (or 5 years in the case of Workforce Development).

Text description – Chart 4

Program name	Active claimants	Former claimants
Skills Training	-$462	-$665
Workforce Development	$68*	n/a

*The impact is not statistically significant over the entire post-participation period.

Note: Impacts are estimated over 4 post-program years (or 5 years in the case of Workforce Development).

Dependence on income support

As shown in Chart 5, active and former EI claimants in Skills Training reduced their dependence on government income supports. The estimates can be interpreted as a change in the probability of receiving EI or SA benefits following participation.

Former claimants in Workforce Development reduced their dependence on government income support however the findings are not statistically significant.

*The impact is not statistically significant over the entire post-program period.

Note: Impacts are estimated over 4 post-program years (or 5 years in the case of Workforce Development).

Text description – Chart 5

Program name	Active claimants (percentage points)	Former claimants (percentage points)
Skills Training	-7.2	-5.3
Workforce Development	-0.9*	n/a

*The impact is not statistically significant over the entire post-participation period.

Note: Impacts are estimated over 4 post-program years (or 5 years in the case of Workforce Development).

Incremental impacts by subgroups of participants

Female participants

Nearly 6,150 EI active and former claimant participants in LMDA programs and services, between 2010 and 2012 in Saskatchewan are female, representing nearly 45% of participants.

The profile of female participants is presented in Table 4 by age, sociodemographic group, and marital status. Information about their educational attainment, occupation and industry is based on the latest job they held prior to applying for EI benefits. Information about sociodemographic groups is self-reported.

Table 4. Profile of female active and former EI claimant participants in EBSMs in Saskatchewan in 2010 to 2012

Categories	Active claimants	Former claimants
Number of participants	2,906	3,237
Age: 30 and under	33%	40%
Age: 31 to 54	58%	54%
Age: 55 and over	9%	6%
Sociodemographic group: Indigenous people	19%	32%
Sociodemographic group: Persons with disabilities	11%	14%
Sociodemographic group: Visible minorities	6%	6%
Sociodemographic group: Recent immigrants	3%	2%
Marital status: Single	42%	54%
Marital status: Married or common-law	38%	27%
Marital status: Widow / divorced / separated	17%	17%
Education or skills level: High school or occupational training	44%	42%
Education or skills level: On-the-job training	18%	25%
Education or skills level: College, vocational education, or apprenticeship training	23%	22%
Education or skills level: University degree	7%	5%

Note: Values may not equal 100% due to rounding or missing information.

Top 3 occupational groups for female participants

Active claimants:

• intermediate sales and service personnel = 20%
• clerical personnel = 17%
• other sales and service personnel = 11%

Former claimants:

• intermediate sales and service personnel = 25%
• other sales and service personnel = 20%
• clerical personnel = 13%

Top 3 industries for female participants

Active claimants:

• retail trade = 15%
• healthcare and social assistance = 13%
• accommodation and food services = 11%

Former claimants:

• accommodation and food services = 18%
• retail trade = 16%
• healthcare and social assistance = 14%

Main findings: Incremental impacts revealed that female active EI claimant participants in Skills Training improved their labour market attachment by increasing in their incidence of employment and earnings and decreasing their dependence on income support. Female active EI claimant participants in Workforce Development improved their labour market attachment through increases in their incidence of employment and earnings. Female former EI claimant participants in Skills Training increased their incidence of employment following program participation. They also decreased their reliance on government income supports due to their decreased use of SA benefits.

Table 5 presents the detailed incremental impacts.

For example, the results revealed that relative to similar female non-participants, active claimants who participated in Skills Training had higher annual average earnings (+$6,202) and incidence of employment (+7.2 percentage points). They also had a lower income support reliance rate (-5.4 percentage points), due to their decreased use of SA benefits (-$559 per year).

Relative to similar female non-participants, active EI claimants who participated in Workforce Development only had a higher annual average incidence of employment (+3.5 percentage points) and employment earnings (+$1,247). The annual average results for EI and SA benefits receipt and dependence on income support are not statistically significant.

Compared to similar female participants in Workforce Development only, former EI claimants in Skills Training had a higher annual average incidence of employment (+6.8 percentage points). The annual average results for employment earnings are not statistically significant. These female participants also had a lower income support reliance rate (-5.7 percentage points) due to their decreased use of SA benefits (-$823 per year).

Table 5. Incremental impacts for female participants

Indicator	Skills Training active claimants	Skills Training former claimants	Workforce Development active claimants
Incidence of employment (percentage points)	7.2***	6.8***	3.5***
Employment earnings ($)	6,202***	2,129	1,247*
EI benefits ($)	198	435**	-60
SA benefits ($)	-559***	-823***	-15
Dependence on income support (percentage points)	-5.4***	-5.7***	-0.6
n=	945	507	1,935

Statistical significance level *** 1%; ** 5%; * 10%, other values are not statistically significant.

Note: Impacts are estimated over 4 post-program years (or 5 years in the case of Workforce Development).

Male participants

Nearly 7,600 EI active and former claimant participants in LMDA programs and services between 2010 and 2012 in Saskatchewan are male, representing about 55% of participants.

The profile of male participants is presented in Table 6 by age, sociodemographic group, and marital status. Information about their educational attainment, occupation and industry is based on the latest job they held prior to applying for EI benefits. Information about sociodemographic groups is self-reported.

Table 6. Profile of male active and former EI claimant participants in EBSMs in Saskatchewan in 2010 to 2012

Categories	Active claimants	Former claimants
Number of participants	3,841	3,741
Age: 30 and under	34%	33%
Age: 31 to 54	56%	59%
Age: 55 and over	10%	8%
Sociodemographic group: Indigenous people	27%	40%
Sociodemographic group: Persons with disabilities	12%	17%
Sociodemographic group: Visible minorities	8%	7%
Sociodemographic group: Recent immigrants	5%	2%
Marital status: Single	52%	58%
Marital status: Married or common-law	35%	25%
Marital status: Widow / divorced / separated	8%	8%
Education or skills level: High school or occupational training	34%	30%
Education or skills level: On-the-job training	31%	37%
Education or skills level: College, vocational education, or apprenticeship training	28%	27%
Education or skills level: University degree	3%	2%

Note: Values may not equal 100% due to rounding or missing information.

Top 3 occupational groups for male participants

Active claimants:

• semi-skilled manual workers = 27%
• other manual workers = 24%
• skilled crafts and trades = 15%

Former claimants:

• other manual workers = 27%
• semi-skilled manual workers = 21%
• skilled crafts and trades = 16%

Top 3 industries for male participants^{Footnote 20}

Active claimants:

• construction = 23%
• manufacturing; and public administration = 11% each
• administrative and support, waste management and remediation services = 7%

Former claimants:

• construction = 22%
• manufacturing; and public administration = 11% each
• retail trade = 8%

Main findings: Incremental impacts revealed that male active EI claimants who participated in Skills Training and Workforce Development improved their labour market attachment through increases in their incidence of employment and employment earnings. Only male active EI claimant participants in Skills Training decreased their dependence on government income support (that is, the combined use of EI and SA benefits). Male active EI claimant participants in Workforce Development decreased their use of EI benefits but increased their use of SA benefits after participation. Male former EI claimants who participated in Skills Training increased their employment earnings and decreased their dependence on income supports (that is, the combined use of EI and SA benefits).

Table 7 presents the detailed incremental impacts.

For example, the results reveal that relative to similar male non-participants, active EI claimants who participated in Skills Training had higher annual average earnings (+$9,839) and incidence of employment (+ 7.9 percentage points). They also decreased their dependence on income supports (-5.9 percentage points), due to their decreased use of SA (-$269 per year) and EI benefits (-$283 per year).

Relative to similar male non-participants, active EI claimants who participated in Workforce Development had higher annual average incidence of employment (+4.5 percentage points) and earnings (+$2,787). They decreased their use of EI benefits (-$357) but increased their use of SA benefits (+$183). Overall, the annual average impact on dependence on government income support is not statistically significant. However, a statistically significant decrease was found in the fifth-year post-program participation (-2.4 percentage points).

Table 7. Incremental impacts for male participants

Indicator	Skills Training active claimants	Skills Training former claimants	Workforce Development active claimants
Incidence of employment (percentage points)	7.9***	0.2	4.5***
Employment earnings ($)	9,839***	3,535***	2,787***
EI benefits ($)	-283*	-108	-357**
SA benefits ($)	-269***	-334***	183**
Dependence on income support (percentage points)	-5.9***	-5.3***	-1.11
n=	1,243	589	2,556

Statistical significance level *** 1%; ** 5%; * 10%, other values are not statistically significant.

Note: Impacts are estimated over 4 post-program years (or 5 years in the case of Workforce Development).

¹ While the annual average impact on dependence on government income supports is not statistically significant, a statically significant decrease for Workforce Development male active claimants was found in year 5 post-program participation (- 2.4 percentage points).

Youth participants

Nearly 4,800 EI active and former claimant participants, between 2010 and 2012, were 30 years of age or younger when they began their program, representing about 35% of participants.

The profile of youth participants is presented in Table 8 by gender, sociodemographic group, and marital status. Information about their educational attainment, occupation and industry are based on the latest job they held prior to applying for EI benefits. Information about sociodemographic groups is self-reported.

Table 8. Profile of youth active and former EI claimant participants in EBSMs in Saskatchewan in 2010 to 2012

Categories	Active claimants	Former claimants
Number of participants	2,297	2,528
Gender: Female	42%	51%
Gender: Male	58%	49%
Sociodemographic group: Indigenous people	21%	33%
Sociodemographic group: Persons with disabilities	9%	11%
Sociodemographic group: Visible minorities	5%	5%
Sociodemographic group: Recent immigrants	2%	2%
Marital status: Single	70%	71%
Marital status: Married or common-law	24%	20%
Marital status: Widow / divorced / separated	4%	5%
Education or skills level: High school or occupational training	39%	36%
Education or skills level: On-the-job training	29%	34%
Education or skills level: College, vocational education, or apprenticeship training	24%	24%
Education or skills level: University degree	3%	2%

Note: Values may not equal 100% due to rounding or missing information.

Top 3 occupational groups for youth participants^{Footnote 21}

Active claimants:

• other manual workers = 21%
• semi-skilled manual workers = 18%
• intermediate sales and service personnel = 13%

Former claimants:

• other manual workers = 18%
• other sales and service personnel; and intermediate sales and service personnel = 16% each
• semi-skilled manual workers = 11%

Top 3 industries for youth participants^{Footnote 22}

Active claimants:

• construction = 19%
• public administration; and retail trade = 10% each
• accommodation and food services = 8%

Former claimants:

• retail trade; construction; and accommodation and food services = 14% each
• public administration = 9%
• health care and social assistance; and manufacturing = 7% each

Main findings: Youth active and former EI claimant participants in Skills Training improved their labour market attachment through increases in their employment earnings and incidence of employment. These youth participants also decreased their dependence on government income support. Youth active EI claimants who participated in Workforce Development improved their labour market attachment through increases in their employment earnings and incidence of employment. The impact on the dependence on government income support is not statistically significant.

Table 9 presents the detailed incremental impacts.

For example, the results reveal that relative to similar youth non-participants, active claimants who participated in Skills Training had higher annual average earnings (+$10,385) and incidence of employment (+7.5 percentage points). They also had a lower income support reliance rate (-4.9 percentage points) due to a decreased use of SA benefits (-$381 per year).

Compared to similar youth participants in Workforce Development only, former EI claimant participants in Skills Training increased their annual average incidence of employment (+5 percentage points) and their employment earning (+$7,634). They also decreased their dependence on government income supports (-7.6 percentage points) through their decreased use of SA benefits (-$900 per year).

Table 9. Incremental impacts for youth participants

Indicator	Skills Training active claimants	Skills Training former claimants	Workforce Development active claimants
Incidence of employment (percentage points)	7.5***	5**	2.7*
Employment earnings ($)	10,385***	7,634***	3,013***
EI benefits ($)	141	269	-149
SA benefits ($)	-381***	-900***	296***
Dependence on income support (percentage points)	-4.9***	-7.6***	0.3
n=	887	429	1,387

Statistical significance level *** 1%; ** 5%; * 10%, other values are not statistically significant.

Note: Impacts are estimated over 4 post-program years (or 5 years in the case of Workforce Development).

Older worker participants

Nearly 640 EI active claimant participants, between 2010 and 2012, were 55 years of age or older when they began their program, representing about 9% of active claimant participants.

The profile of older worker participants is presented in Table 10 by gender, sociodemographic group, and marital status. Information about their educational attainment, occupation and industry are based on the latest job they held prior to applying for EI benefits. Information about sociodemographic groups is self-reported.

Table 10. Profile of older worker active EI claimant participants in EBSMs in Saskatchewan in 2010 to 2012

Categories	Active claimantsFootnote 23
Number of participants	633
Gender: Female	41%
Gender: Male	59%
Sociodemographic group: Indigenous people	15%
Sociodemographic group: Persons with disabilities	13%
Sociodemographic group: Visible minorities	6%
Sociodemographic group: Recent immigrants	3%
Marital status: Single	25%
Marital status: Married or common-law	48%
Marital status: Widow / divorced / separated	23%
Education or skills level: High school or occupational training	42%
Education or skills level: On-the-job training	20%
Education or skills level: College, vocational education, or apprenticeship training	25%
Education or skills level: University degree	6%

Note: Values may not equal 100% due to rounding or missing information.

Top 3 occupational groups for older worker active claimant participants^{Footnote 24}:

• semi-skilled manual workers = 22%
• clerical personnel; other manual workers; intermediate sales and service personnel; and other sales and service personnel = 10% each
• skilled crafts and trades = 9%

Top 3 industries for older worker active claimant participants^{Footnote 25}:

• construction = 13%
• retail trade; and public administration = 10% each
• manufacturing; and administrative and support, waste management and remediation services = 8% each

Main findings: Incremental impacts revealed that older workers in Workforce Development improved their labour market attachment by increasing their incidence of employment and employment earnings.

Table 11 presents the detailed incremental impacts.

For example, the results revealed that compared to similar older worker non-participants active claimants in Workforce Development had higher annual earnings (+$3,615) and incidence of employment (+8.4 percentage points). The impact on the dependence on income support including EI and SA benefits is not statistically significant.

Table 11. Incremental impacts for older worker participants in Workforce Development

Indicator	Workforce Development active claimants
Incidence of employment (percentage points)	8.4**
Employment earnings ($)	3,615**
EI benefits ($)	483
SA benefits ($)	265
Dependence on income support (percentage points)	-0.1
n=	453

Statistical significance level *** 1%; ** 5%; * 10%, other values are not statistically significant.

Note: Impacts are estimated over 4 post-program years (or 5 years in the case of Workforce Development).

Indigenous participants

Nearly 4,200 EI active and former claimant participants, between 2010 and 2012, self-identify as being Indigenous Canadians, representing about 31% of participants.

The profile of Indigenous participants is presented in Table 12 by gender, age, and marital status. Information about educational attainment, occupation and industry are based on the latest job held prior to applying for EI benefits.

Table 12. Profile of Indigenous active and former EI claimant participants in EBSMs in Saskatchewan in 2010 to 2012

Categories	Active claimants	Former claimants
Number of participants	1,617	2,566
Gender: Women	35%	41%
Gender: Men	65%	59%
Age: 30 and under	30%	33%
Age: 31 to 54	64%	62%
Age: 55 and over	6%	5%
Marital status: Single	58%	62%
Marital status: Married or common-law	26%	22%
Marital status: Widow / divorced / separated	11%	9%
Education or skills level: High school or occupational training	35%	34%
Education or skills level: On-the-job training	30%	33%
Education or skills level: College, vocational education, or apprenticeship training	26%	25%
Education or skills level: University degree	5%	4%

Note: Values may not equal 100% due to rounding or missing information.

Top 3 occupational groups for Indigenous participants^{Footnote 26}

Active claimants:

• semi-skilled manual workers = 21%
• other manual workers = 20%
• other sales and service personnel; and skilled crafts and trades workers =11% each

Former claimants:

• other manual workers = 20%
• semi-skilled manual workers = 15%
• other sales and service personnel = 14%

Top 3 industries for Indigenous participants^{Footnote 27}

Active claimants:

• public administration = 22%
• construction = 19%
• retail trade = 7%

Former claimants:

• public administration = 18%
• construction = 15%
• retail trade; and accommodation and food services = 9% each

Main findings: Indigenous participants improved their labour market attachment through increases in their incidence of employment and employment earnings. Active and former claimants in Skills Training decreased their dependence on government income support mostly through decreasing their use of SA benefits. The findings are consistent with findings from the Indigenous Skills and Employment Training Strategy evaluation.

Table 13 presents the detailed incremental impacts.

For example, the results reveal that relative to the comparison group, Indigenous active claimants in Skills Training had higher annual earnings (+$9,388 per year) and incidence of employment (+11.5 percentage points). They also had a lower income support reliance rate (-4.4 percentage points), mostly due to decreasing their use of SA benefits (-$316 per year).

Indigenous former claimants in Skills Training had higher annual earnings (+$5,242 per year) and incidence of employment (+6. percentage points). They also had a lower income support reliance rate (-6.7 percentage points), mostly due to decreasing their use of SA benefits (-$792 per year).

Table 13. Incremental impacts for Indigenous participants

Indicator	Skills Training active claimants	Skills Training former claimants	Workforce Development active claimants
Incidence of employment (percentage points)	11.5***	6***	3.3*
Employment earnings ($)	9,388***	5,242***	3,464***
EI benefits ($)	733**	198	-126
SA benefits ($)	-316**	-792***	366***
Dependence on income support (percentage points)	-4.4**	-6.7***	0.9
n=	470	460	1,106

Statistical significance level *** 1%; ** 5%; * 10%, other values are not statistically significant.

Note: Impacts are estimated over 4 post-program years (or 5 years in the case of Workforce Development).

Skills Training participants in Regina/Saskatoon

Nearly 1,000 EI active and former claimant participants in Skills Training between 2010 and 2012 in Saskatchewan were in Regina/Saskatoon, representing about 7% of participants.

Main findings: Participants in Regina/Saskatoon improved their labour market attachment through increases in their incidence of employment and employment earnings. They also decreased their dependence on government income support mostly by decreasing their use of SA benefits.

Table 14 presents the detailed incremental impacts.

For example, the results reveal that relative to the comparison group, Regina/Saskatoon active claimants in Skills Training had higher annual earnings (+$6,616 per year) and incidence of employment (+7.7 percentage points). They also had a lower income support reliance rate (-6 percentage points), mostly due to decreasing their use of SA benefits (-$682 per year).

Regina/Saskatoon former claimants in Skills Training had higher annual earnings (+$7,479 per year) and incidence of employment (+6.6 percentage points). They also had a lower income support reliance rate (-6.5 percentage points), mostly due to decreasing their use of SA benefits (-$708 per year).

Table 14. Incremental impacts for Skills Training participants in Regina/Saskatoon

Indicator	Skills Training active claimants	Skills Training former claimants
Incidence of employment (percentage points)	7.7***	6.6**
Employment earnings ($)	6,616***	7,479***
EI benefits ($)	178	191
SA benefits ($)	-682***	-708***
Dependence on income support (percentage points)	-6***	-6.5***
n=	698	289

Statistical significance level *** 1%; ** 5%; * 10%, other values are not statistically significant.

Note: Impacts are estimated over 4 post-program years.

Skills Training participants outside of Regina/Saskatoon

Nearly 2,300 EI active and former claimant participants in Skills Training between 2010 and 2012 in Saskatchewan were outside of Regina/Saskatoon, representing about 17% of participants.

Main findings: Active claimant participants outside of Regina/Saskatoon improved their labour market attachment through increases in their incidence of employment and employment earnings. They also decreased their dependence on government income support mostly by decreasing their use of SA benefits. Former claimant participants outside of Regina/Saskatoon improved their incidence of employment and decreased their dependence on government income support mostly by decreasing their use of SA benefits.

Table 15 presents the detailed incremental impacts.

For example, the results revealed that relative to the comparison group, active claimant participants in Skills Training outside of Regina/Saskatoon had higher annual earnings (+$9,349 per year) and incidence of employment (+8.4 percentage points). They also had a lower income support reliance rate (-4.8 percentage points), mostly due to decreasing their use of SA benefits (-$279 per year).

Former claimant participants in Skills Training outside of Regina/Saskatoon had higher incidence of employment (+3.6 percentage points). They also had a lower income support reliance rate (-5.2 percentage points), mostly due to decreasing their use of SA benefits (-$581 per year). However, former claimants increased their use of EI benefits (+$351) as they are likely to requalify for EI benefits after participation due to their stronger labour market attachment.

Table 15. Incremental impacts for Skills Training participants outside of Regina/Saskatoon

Indicator	Skills Training active claimants	Skills Training former claimants
Incidence of employment (percentage points)	8.4***	3.6**
Employment earnings ($)	9,349***	1,987
EI benefits ($)	-131	351**
SA benefits ($)	-279***	-581***
Dependence on income support (percentage points)	-4.8***	-5.2***
n=	1506	804

Statistical significance level *** 1%; ** 5%; * 10%, other values are not statistically significant.

Note: Impacts are estimated over 4 post-program years.

Cost-benefit analysis

This analysis is based on the EBSM medium-term incremental impacts previously described in this report. Costs and benefits are examined over the participation period of 1 or 2 years and 5 or 10 years after the end of participation.^{Footnote 28}

The cost-benefit analysis addresses the following questions:

1. Are the benefits from EBSMs exceeding the costs within 5 years (for Workforce Development), 10 years (for Skills Training) after the end of participation?
2. How much is the benefit for the government and society if the government spends $1 in EI part II funding?
3. How many years does it take the benefits to recover the costs?

The following results are presented from the social perspective, that is, the government and individual combined. This allows for a sound assessment of program effectiveness in achieving its objectives of helping unemployed individuals to obtain and maintain employment and to generate EI savings.

Table 16 presents the cost-benefit results for active and former EI claimant participants.

Table 16. Cost-benefit results for active and former EI claimant participants

Variable	Skills Training active claimants (10 years post-program)	Workforce Development active claimants (5 years post-program)	Skills Training former claimants (10 years post-program)
Net present value	$74,508	$10,569	$32,675
Benefit-cost ratio	$15.53	$15.93	$7.02
Payback period (years after end of participation)	1.2	1.4	1.0
Social return	1,453%	1,493%	602%
Savings to public health care	$695	$103	$437

The information above provides examples of the net present value, the benefit-cost ratio, the payback period, the social rate of return and savings to health care costs.

Skills Training^{Footnote 29}

During the 2010 to 2012 period, Skills Training represents almost 73% of EBSM expenditures under the LMDAs in Saskatchewan. The average duration of a Skills Training Action Plan Equivalent is 39 weeks for active claimants and 38 weeks for former claimants.

As shown in Table 16, over the 10-year post-program period the benefit for active claimants is +$74,508 higher than the costs, yielding a social return on investment of 1,453%. This means that if the government spends $1 on Skills Training for active EI claimants, it generates +$15.53 of benefit for society. It takes 1.2 years for the benefits to recover the costs of programming. Overall, there is a savings to health care costs of $695 per participant.

The benefit for former claimants is +$32,675 higher than the costs, yielding a social return of 602% on investment. This means that if the government spends $1 on Skills Training for active EI claimants, it generates +$7.02 of benefit for society. It takes 1 year for the benefits to recover the costs of programming. Overall, there is a savings to health care costs of $437 per participant.

Workforce Development^{Footnote 30}

Workforce Development includes a variety of services such as computer access for job search services, group sessions to prepare for an interview, career counselling, and action plan development. The administrative data, however, do not allow to identify what proportion of Workforce Development interventions belong to each category or the intensity of services offered to participants.

While Workforce Development is often provided with other EBSMs, this analysis examined only participants who received one or more Workforce Development services without participating in other EBSMs. Workforce Development represents about 15% of total EBSM expenditures in Saskatchewan between 2010 and 2012. The average length of a Workforce Development-only Action Plan Equivalent is 12 weeks compared to between 33 to 49 weeks for active EI claimant participants in other EBSMs.

As shown in Table 16, over the 5-year post-program period the benefits for active claimants in Workforce Development are $10,569 higher than the costs, yielding a social return on investment of 1,493%. This means that if the government spends $1 on Workforce Development for active EI claimants, it generates +$15.93 of benefit for society. It takes 1.4 years after participation for the benefits to recover the costs. Overall, there is a savings to health care costs of $103 per participant.

Supplemental studies

Self-Employment^{Footnote 31}

Program design and delivery

The Self-Employment program aims to assist participants in creating employment for themselves by providing them with a range of services including:

• assistance with business plan development
• counselling, coaching, and mentoring
• entrepreneurial training and workshops

In addition to being unemployed and EI-eligible, participants must not already own and/or operate their businesses prior to program participation.

Saskatchewan has the flexibility to design and deliver the program to meet its labour market needs. In fall 2018, the program was delivered mainly through third-party organizations, including such as:

• business development corporations
• Community Futures (used in rural areas)

Program officials reported that the amount allocated to the Self-Employment program is influenced by demand for the program and service delivery costs. The application process is structured and aimed to ensure that participants are suited for self-employment, have a viable business idea and the financial resources to launch a business.

Participants’ employment outcomes^{Footnote 32}

Businesses survival rates and success factors

Among the 31 respondents who started a business, 21 or 68% of them were still operating their business at 2 to 4 years post-program. The business survival rate is positive compared to a 2018 Statistics Canada study^{Footnote 33} that found that less than half of unincorporated self-employed individuals continued operations for more than 2 years.

Regarding factors influencing the success or failure of self-employment businesses:

• participants who started a business and were still in operation at the time of survey attributed their business success to:
  • their dedication, hard work and positive attitude
  • their own abilities, flexibility, and adaptability
  • their network and business contacts
  • their own marketing and networking abilities
  • having good credit
  • being in a good financial position
• participants who started a business but were forced to close it or participants who did not launch a business attributed the closure to:
  • failure to adapt to having a self-employment business
  • changes in personal circumstances
  • finding another job
  • lack of supports

Earning outcomes and reliance on income support

Survey respondents were not comfortable answering questions that related to their earnings. This situation made it difficult to compare the pre- and post-earnings of Self-Employment participants.

As a complement to the earning questions, survey respondents did assess their financial well-being. When considering their entire financial situation:

• 70% of respondents said that they are financially about the same or better off after the program
• 68% of respondents said that their household net worth is about the same or higher after the program

In line with survey findings, 6 key informants state that immediate increases in earnings are not necessarily an expected outcome of the program. 

Regarding the reliance on government income support, participants reduce reliance on the use of EI and SA following program participation.

Satisfaction with services received and current employment

Fifty three percent (53%) of respondents who started a self-employment business report that they are more satisfied with their job situation after program participation.

The survey did examine the contribution of the program to the success of self-employment businesses. Over 95% of survey respondents who launched a self-employment business rate the following services and training as very or somewhat important to the business launch, operation, and success:

• orientation or information sessions
• assessment of entrepreneurial suitability
• financial assistance for business start-up costs
• one-on-one mentorship
• living allowance during participation
• training on operating a business
• assistance with developing a business plan

Challenges and lessons learned related to program design and delivery

Key informants identify the following challenges related to program design and delivery, including:

• impractical and outdated program eligibility criteria
• a lack of program awareness
• insufficient financial support for participants

Best practices related to program design and delivery included:

• having knowledgeable service providers and being responsive to their feedback
• providing practical, personalized training that is responsive to participants’ needs
• qualified and experienced business coaches delivering training
• employing a thorough participant selection process

Key considerations for Self-Employment program and policy development

The following considerations emerged as part of the study.

The Self-Employment program can benefit from an updated objective specifying that it is dedicated to eligible participants who have a viable business idea, the financial or in-kind resources to launch a business, and the required level of dedication.

The data collection process should include only participants who have been deemed suitable for self-employment and accepted into the program. This will require excluding candidates who attended information sessions alone or those deemed not suited for self-employment. The latter participants can be reported under Workforce Development.

Indicators of program success can include: increase in employment and/or self-employment levels; medium-term increase in earnings; business survival rate similar to the local economy and/or the sector; and acquisition of transferable skills.

Saskatchewan may wish to consult with their service delivery network on the extent to which identified challenges are applicable to their unique context, and how best to address them along with integrating lessons learned that can benefit program delivery.

Rationale

The Self-Employment program aims to assist participants in creating employment for themselves. The participant’s application process is structured and aimed to ensure that they are suited for self-employment, have a viable business idea, and the financial resources to launch a business. However, the survey revealed that:

• 30% of participants did not launch a business
• of those who did launch a business, nearly a third were unable to maintain the operation of the business they started as part of the program
• 5 survey respondents confirmed that they did not participate in the program

Key informants identified a variety of factors that that support the success of participants at starting and maintaining their businesses, including: desire/motivation/drive/tenacity, adaptability/flexibility, being a part of a network of entrepreneurs, ability to market and network oneself, having good credit/financial position, being skilled in marketing and entrepreneurship, having good accounting practices, listening to counsel/advice, having a solid business plan, having family support; and understanding of the market.

Key informants identified a variety of reasons that participants fail at starting and maintaining their businesses, including: failing to adapt, changes in personal circumstances, taking paid employment elsewhere, lack of coach-ability, lack of support, unforeseen delays in getting the business operational, increased competition, being in a tenuous financial position, insufficient research on market demand, and unrealistic expectations.

The survey confirmed that participants acquire transferable skills through training and workshops, they experience increase in employment and medium-term earnings, and they create additional jobs. As well, business survival rates mirror those observed for small business in the economy. These indicators are useful in measuring and reporting program success as well as managing contribution agreements with service providers.

Skills Training-Apprentices

The objective of the program is to help apprentices become skilled tradespeople and to increase their labour market attachment. Program participants have generally chosen a career and are already attached to the labour market. The apprenticeship process involves on-the-job learning and technical training in a classroom setting.

Apprentices who have worked enough hours to qualify for EI can apply to receive EI Part I benefits while on training. The program provides financial assistance to EI eligible apprentices to help them offset the costs they incur while they attend technical training. The level of funding is based on the needs of apprentices, the location of the training, and any fees paid by the apprentices.^{Footnote 34}

The profile of program participants is presented in Table 17 by gender, sociodemographic group, age and marital status. Information about educational attainment, occupation and industry is based on the last job held prior to applying for EI benefits. Information about sociodemographic groups is self-reported.

Table 17. Profile of active and former EI claimant participants in Skills Training-Apprentices in Saskatchewan in 2010 to 2012

Categories	Active claimants	Former claimants
Number of participants	2,979	910
Gender: Female	5%	12%
Gender: Male	95%	88%
Age: 30 and under	83%	73%
Age: 31 to 54	17%	26%
Age: 55 and over	0%	❮10 participants
Sociodemographic group: Indigenous people	9%	18%
Sociodemographic group: Persons with disabilities	4%	6%
Sociodemographic group: Visible minorities	4%	5%
Sociodemographic group: Recent immigrants	2%	1%
Marital status: Single	71%	59%
Marital status: Married or common-law	25%	34%
Marital status: Widow / divorced / separated	3%	4%
Education or skills level: High school or occupational training	2%	13%
Education or skills level: On-the-job training	7%	21%
Education or skills level: College, vocational education or apprenticeship training	91%	63%
Education or skills level: University degree	❮10 participants	❮10 participants

Top 3 occupational groups for Skills Training-Apprentices

Active claimants:

• skilled crafts and trades workers = 87%
• other manual workers = 6%
• semi-professionals and technicians= 2%

Former claimants:

• skilled crafts and trades workers = 56%
• other manual workers = 18%
• semi-skilled manual workers = 8%

Top 3 industries for Skills Training-Apprentices

Active claimants:

• construction = 59%
• other services (excluding public administration) = 8%
• manufacturing = 7%

Former claimants:

• construction = 38%
• manufacturing = 12%
• other services (excluding public administration) = 8%

Labour market outcomes

The labour market outcomes are based on individuals who began their participation during the 2010 to 2012 period. Statistics focus on 5 years before program participation and 5 years after the program start year.

Active claimants

As shown in Chart 6, program participants increase their average earnings from $18,590 in the fifth year pre-program to $64,293 in the fifth year after the program start year.

Text description – Chart 6

Year pre-post participation	Employment earnings for active claimants
5 years pre-program	$18,590
4 years pre-program	$20,741
3 years pre-program	$22,803
2 years pre-program	$27,092
1 year pre-program	$35,263
Program start year	$36,333
1-year post-program	$45,569
2 years post-program	$51,562
3 years post-program	$58,033
4 years post-program	$63,836
5 years post-program	$64,293

The proportion of employed participants declines by just under 2 percentage points on average after the program start year but remains around 96% on average during the post program years. The proportion of participants on EI Part I decreases from 100% in the program start year to 25% in the fifth year after the program start year. Participants decrease their dependence on income support from 17% in the program start year to 4% in the fifth year after participation.

Former claimants

As shown in Chart 7, program participants increased their average earnings from $20,900 in the fifth year pre-program to $58,137 in the fifth year after the program start year.

Text description- Chart 7

Year pre-post participation	Employment earnings for former claimants
5 years pre-program	$20,900
4 years pre-program	$24,215
3 years pre-program	$26,391
2 years pre-program	$28,994
1 year pre-program	$34,943
Program start year	$41,680
1-year post-program	$48,578
2 years post-program	$54,122
3 years post-program	$57,636
4 years post-program	$58,168
5 years post-program	$58,137

The proportion of employed participants declined by 4 percentage points between the pre and post program years. The percentage of employed participants remains at 92% on average in the 5 post-program years. The proportion of participants on EI Part I decreased from 62% in the program start year to 25% in the fifth year after the program start year. Participants decreased their dependence on income support from 11% in the program start year to 8% in the fifth year after participation.

Conclusions and recommendations

Overall, incremental impacts demonstrate that participation in most EBSMs improves labour market attachment and reduces dependence on government income supports compared to similar non-participants. A subgroup analyses shows that with some exceptions, Skills Training improves the labour market attachment, employment earnings and reduced the dependence on income support for most subgroups of participants. Workforce Development alone was found to improve the labour market attachment and employment earnings for female, male, youth, older workers, and Indigenous participants. As well, the social benefits of participating in EBSMs exceeds the costs of investments for most interventions over time.

Recommendations

Since 2012, 15 qualitative and quantitative studies addressed issues and questions related to EBSM design, delivery and effectiveness. The quantitative studies successfully assessed the effectiveness and efficiency of EBSMs by producing incremental impacts and cost-benefit analysis. The qualitative studies help to contextualize the findings from the quantitative studies and to identify specific challenges, lessons learned, and best practices associated with the design and delivery of EBSMs. Each study included key considerations for program and policy development or recommendations.

The recently completed evaluation of the Workforce Development Agreements complements the LMDA qualitative studies. This evaluation was also supported by literature reviews and provided unique insights into challenges and lessons learned to assist persons with disabilities, immigrants and those further removed from the labour market.

Most results from this evaluation stem from the conduct of advance causal analysis whereby impacts found could be attributed to a specific EBSM. These analyses are predicated on having access to high quality administrative records, thereby confirming the importance of the capacity to leverage and integrate relevant administrative data.

From these main findings, 2 key recommendations emerge.

Recommendation #1: ESDC and Saskatchewan are encouraged to share and discuss lessons learned, best practices and challenges associated with the design and delivery of EI-funded programming. Discussions are encouraged at the bilateral or multilateral levels as well as with service delivery network if necessary.

Recommendation #2: ESDC and Saskatchewan are encouraged to pursue efforts to maintain and strengthen data collection provisions in support of reporting, performance measurement and data-driven evaluations at the national and provincial levels.

References

Allin, S., Corscadden, L., Gapanenko, K., & Grignon, M. Lifetime Distributional Effects of Publicly Financed Health Care in Canada. CIHI, 2013.

Douwere G. and Huju Liu. The Entry into and Exit out of Self-employment and Business Ownership in Canada. Statistics Canada, 2018.

Employment and Social Development Canada. Cost-Benefit Analysis: Incorporating Public Health Care Costs Savings in the Context of the Labour Market Programs Evaluation. Internal document, 2022. Available upon request.

Employment and Social Development Canada. Cycle II of the Evaluation of the Labour Market Development Agreements: Cost-Benefit Analysis of Employment Benefits and Support Measures. Internal document, 2015. Available upon request.

Employment and Social Development Canada. Employment Insurance Monitoring and Assessment Reports. 2011 to 2022.

Employment and Social Development Canada. Evaluation of the Labour Market Development Agreements, Design and delivery of the Self-Employment program in Saskatchewan. Internal report, 2020.

Employment and Social Development Canada. Labour Market Development Agreements Process for Determination of Similarity. Internal document, 2012. Available upon request.

Employment and Social Development Canada. Technical Report on the Analysis of Employment Benefits and Support Measures (EBSMs) Profile, Outcomes and Medium-Term Incremental Impacts from 2010 to 2017. Internal report, 2021. Available upon request.

Employment and Social Development Canada. Third Cycle for the Horizontal Evaluation of the Labour Market Development Agreements: Quantitative Methodology Report. Internal document, 2019. Available upon request.

Appendix A. List of 5 studies included in this synthesis report

Study 1: Examination of medium-term outcomes from 2010 to 2017

Evidence generated: provincial level profile of active and former EI claimants; outcomes by claimant type and by subgroup

Methods: before and after results of program participation

Reference period: 2010 to 2012 participants

Observation period: up to 12 years (5 years before participation, 1 to 2 years of participation, and 5 years after participation

Study 2: Estimation of medium-term incremental impacts from 2010 to 2017

Evidence generated: incremental impacts for active and former EI claimants; incremental impacts by subgroup; profile and socio-demographic characteristics of participants

Methods: non-experimental method using propensity score matching in combination with Difference-in-Differences; statistical profiling

Reference period: 2010 to 2012 participants

Observation period: Up to 7 years (1 to 2 years in program, and 5 years after participation)

Study 3: Cost-Benefit Analysis of Employment Benefits and Support Measures in Saskatchewan

Evidence generated: cost-benefit analysis

Methods: non-experimental method using propensity score matching in combination with Difference-in-Differences; cost analysis

Reference period: 2010 to 2012 participants

Observation period: 5 years post-program for Workforce Development; 10 years post-program for Skills Training

Study 4: Cost-Benefit Analysis: Incorporating Public Health Care Costs Savings in the Context of the Labour Market Programs Evaluation in Saskatchewan

Evidence generated: Cost-benefit analysis

Methods: Estimation of adjusted annualized healthcare costs

Reference period: 2010 to 2012 participants

Observation period: 5 years post-program for Workforce Development; 10 years post-program for Skills Training

Study 5: Design and delivery of the Self-Employment program in Saskatchewan

Evidence generated: program design, delivery and success; define outcomes attributed to the program; fill in knowledge gaps; challenges and lessons learned

Methods: document review, statistical analysis of administrative data, Canadian self-employment literature and statistics, 6 semi-structured telephone interviews with 7 key informants, survey of Self-Employment participants in Saskatchewan

Reference period: 2015 to 2017 participants

Observation period: 2015 to 2020