Technical Report: Estimating the Net Impact of the Opportunities Fund for Persons with Disabilities Program

On this page

• Glossary
• Acknowledgement
• Abstract
• 1. Introduction
• 2. Program overview
• 3. Methodology
  • 3.1 Data and unit of analysis
  • 3.2 Estimating the net impact
• 4. Socio-demographic profile of the participants
• 5. Net impact analysis results
  • 5.1 OF overall
  • 5.2 Enhanced employment assistance services
  • 5.3 OF-skills for employment
  • 5.4 Wage subsidy
  • 5.5 Work experience
• 6. Cost benefit analysis
• 7. Conclusion
• References
• Appendix A: Detailed results on socio-demographic characteristics
• Appendix B: Net impacts results
• Appendix C: Distribution of propensity score before and after matching

Glossary

Conditional independence assumption

The conditional-independence assumption requires that the common variables that affect participation assignment and participation-specific outcomes be observable.

Counterfactual

The counterfactual refers to what would have happened to the participants in the absence of the program.

Difference-in-differences

The difference-in-differences (or "double difference") estimator is defined as the difference in average outcome in the participant group before and after participation minus the difference in average outcome in the control group before and after participation.

Discount rate

A discount rate is used to calculate the Net Present Value (NPV) which is the value of all future cash flows over the entire life of an investment discounted to the present.

Inverse probability weighted

IPW use weighted averages of the observed outcome variable to estimate the potential-outcome means. The weights are the reciprocals of the participation probabilities estimated by a participation model.

Kernel matching

Kernel matching is a non-parametric matching estimator that uses weighted averages of all individuals in the comparison group to construct the counterfactual outcome.

Nearest neighbor matching

Nearest-neighbor matching uses the distance between observed variables to find similar individuals.

Propensity score matching

The propensity score is the conditional probability of participating in the intervention given the pre-participation variables. Propensity score matching uses the distance between estimated propensity scores to find similar individuals.

MSCPF

The Marginal Social Cost of Public Funds (MSCPF) represents the loss incurred by society when raising additional revenues, such as taxes, to fund government spending.

Social return

Social rate of return is the ratio between the net benefit to society and the direct program costs to the government. In other words, it shows the dollar value of the net benefit to the society when the government spends $1 on a program.

Acknowledgement

We would like to express our appreciation to those who contributed to this technical report and provided valuable information on the operations of the Opportunities Fund for Persons with Disabilities program. We gratefully acknowledge the valuable inputs and comments received from Jérôme Mercier, Laura MacFadgen, Donna Smith-Moncrieffe and Richard R De Marco. We would also like to thank our colleagues Eppo Maertens and Khaddouj Souaid from the Skills and Employment Branch for their collaboration and assistance.

Abstract

This technical report presents the methods and techniques employed to measure the net impact analysis and the cost-benefit analysis of the Opportunities Fund (OF) for persons with disabilities in Canada^{Footnote 1}. The analysis uses rich longitudinal administrative data from Employment and Social Development Canada’s Labour Market Program Data Platform for individuals who participated in the OF program from January 2011 to December 2012. This report applies propensity score matching combined with the difference-in-differences methodology to estimate the net impact of the OF program on participants 5-years post participation until 2017. The report's main strength is its reliance on the rich administrative data, which allows for controlling numerous socio-demographic and labour market characteristics in the propensity score model. The techniques used to build the counterfactual comparison group effectively matched participants to similar non-participants for each intervention. In addition, this report uses rigorous statistical tests using other estimation methods to check the robustness of the results. A cost-benefit analysis assesses the cost effectiveness of the OF program by estimating the net social benefit from this program. These methods and techniques generated robust evaluation results, demonstrating that the OF program is effective in improving the labour market attachment of program participants. Overall, the OF program yields a positive social return on investment over the ten-year post participation.

1. Introduction

The Government of Canada invests in specific programs to facilitate the participation of persons with disabilities in the labour market. Over the past 2 decades, Employment and Social Development Canada (ESDC) has funded organizations known as Community Coordinators to provide labour market programming for persons with disabilities. This technical report presents the methods and findings from the net impact analysis and cost-benefit analysis employed in phase 2 of the 3rd summative evaluation^{Footnote 2} of the Opportunities Fund program for persons with disabilities.

Launched by ESDC in 1997, the Opportunities Fund for Persons with Disabilities (OF) Program is designed to fill a gap in federal supports for persons with disabilities who are not eligible for Employment Insurance program or training support under the Employment Insurance Act Part II. The program's objective is to assist these individuals in preparing for, obtaining and keeping employment or becoming self-employed, thereby increasing their economic participation and independence. This report examines the following interventions under the Opportunities Fund:

• Enhanced employment assistance services
• Skills for employment
• Wage subsidies
• Work experience

This technical report examines the effectiveness and efficiency of the OF program through an analysis of the employment-related outcomes of participants over a 10-year period from 2007 to 2017 (5 year pre-participation and 5 year post-participation) using non-experimental methods. This report presents the techniques used to estimate the net impacts of the program via propensity score kernel matching^{Footnote 3} combined with the difference-in-differences method. The econometric model used in this report builds on rich information about an individual’s labour market experience, including socio-demographic characteristics as well as labour market history. The report uses data from the Labor Market Program Data Platform, which is an integrated, longitudinal administrative database. This analysis examines all individuals who participated in the OF program between January 2011 and December 2012. The comparison group contains of individuals who meet the same eligibility criteria and socio-demographic characteristics as OF participants, but had received only minor interventions via Employment Assistance Services as part of the Labour Market Development Agreements (LMDA) and self-declared as having a disability. This report also presents the results from the cost-benefit analysis, which addresses the economic costs and benefits arising from interventions delivered under the OF program.

The net impact analysis indicates that relative to the comparison group^{Footnote 4}, OF participants experienced higher earnings and incidence of employment, as well as a small decrease in their reliance on social assistance benefits. Cost-benefit analysis suggests that the OF program yields economic benefits that outweigh the government’s investment in the program. From a societal perspective, it takes less than 4 years to recover the cost of the program. Skills for Employment has the highest social return among all types of interventions.

This report is organized as follows. Section 2 presents a brief overview of the OF program. Section 3 describes the methodology used to analyze the effectiveness of the program, and Section 4 presents the socio-demographic profile of the participants. Section 5 and 6 present the net impact and cost-benefit analysis respectively. Section 7 provides concluding remarks.

2. Program overview

Launched in 1997, the OF program was designed to fill a gap in federal supports for persons with disabilities who are not eligible for the Employment Insurance (EI) program or training support under EI Part II. The OF program aims to assist participants integration in the labor market by providing different types of services including job search supports, pre-employability services, wage subsidies, work placements, and employer awareness initiatives to encourage employers to hire persons with disabilities. It delivers 7 types of interventions, namely:

• Enhanced Employment Assistance Services (OF-EAS)
• Skills for Employment (OF-SFE)
• Wage Subsidies (OF-WS)
• Work Experience (OF-WE)
• Self-Employment, Developmental and Services

However, Self-Employment, Developmental and Services interventions are not considered due to the insufficient number of participants to conduct the net impact analysis. A brief description of the 4 intervention types considered in this report are provided below.

Enhanced employment assistance services
This intervention supports a mixture of pre-employment activities tailored to meet the needs of persons with disabilities and facilitate their integration into employment. Broadly categorized, these activities include:

• counselling
• job placement services
• access to labour market information
• assistance in determining appropriate skills training

Skills for employment
This intervention enables participants to undertake short-duration training and develop skills, ranging from basic to advance. Such skills must relate to a specific job opportunity where there is a reasonable expectation of employment in a stable or expanding sector.

Wage subsidy
This intervention provides a financial incentive to encourage employers to hire persons with disabilities (whom they would not normally hire), in the form of a share of the wages paid to these participants. Such activities place participants in positions that are likely to lead to permanent jobs with the host employers, and to assist participants whose likelihood of finding employment will markedly improve through the creation of employer-employee relationships.

Work experience
This intervention provides the same training as for the Wage Subsidy intervention except that the positions occupied by participants in “Work Experience” activities are only formed for the purposes of the Program. Therefore, there was not the same expectation that the host employer would retain the participant after the subsidy ended. After April 2015, this intervention was no longer eligible for funding.

The OF program gives funding to support a range of projects. This is to ensure that eligible persons with disabilities can access the assistance needed to integrate or re-integrate into the labour market. The vast majority of funding for the OF activities is delivered through agreements between ESDC/Service Canada and contribution recipients who act as community coordinators for the program^{Footnote 5}. Community Coordinators, also referred to as funding or contribution recipients, act as intermediaries between participants and employers.

Reforms introduced in budgets 2012, 2013 and 2014 resulted in the following operational changes:

• connect more Canadians with disabilities to work experience in small and medium-sized businesses
• provide demand-driven training solutions for persons with disabilities and make the program more responsive to labour market needs
• involve employers and community organizations in project design and delivery
• ensure employers are aware of the invaluable contribution of persons with disabilities

3. Methodology

3.2 Estimating the net impact

The methodology of this report is built on the same framework as the evaluation of the Labour Market Development Agreements. The procedure for estimating net impacts is based on a non-experimental approach that measures the effectiveness of the program by comparing the actual outcomes for the participants in different interventions to their counterfactual outcomes (that is, the outcomes they would have experienced in the absence of an intervention). Because a given individual cannot be both a participant and a non-participant in an OF intervention during the reference period, an appropriate comparison group is needed to “stand in” as the counterfactual. Based on the available data, we built our comparison group using a subset of individuals who have the same characteristics as OF participants, but had received only low intensity service through minor Employment Assistance Services interventions as part of the Labour Market Development Agreements.

The following sections discuss the approaches for determining the counterfactual comparison group and implementing the propensity score matching combined with difference-in-differences methodology.

3.2.1 Selection of comparison group

This report uses a non-experimental method to determine the counterfactual comparison group. Non-experimental methods generate comparison groups that are akin to the group of program participants by using techniques, such as propensity score matching combined with difference in differences. This approach avoids ethical and political challenges of an experimental design that would require creating a comparison group of non-participants that are randomly denied access to a program.

The main challenge of assessing the net impact of participation in a program is selecting a comparison group that closely resembles the program participants. The comparison group for the OF program is composed of individuals who shared the same eligibility criteria and socio demographic characteristics as OF participants, but had received only minor interventions via LMDA-EAS and self-declared as having a disability. The comparison group chosen are from this low intensity group who started participation during the same reference period from January 2011 to December 2012. Using LMDA-EAS participants as the comparison group has several advantages:

• the universe of LMDA-EAS cases is very large so it is possible to obtain close matches to program participants both in terms of their observable characteristics and in terms of the timing of their participation
• numerous pieces of information on LMDA-EAS participants facilitate matching techniques needed to develop the counterfactual for OF participants (for example type of disability)
• EAS interventions are low-intensity and of short duration, representing a true “limited treatment” comparison group

3.2.2 Implementation of the matching estimator combined with difference-in-differences method

We used non-experimental propensity score based weighting approach to measure the program impacts. This approach aims to ensure that the participants and comparison group are balanced in terms of background factors (that is, sociodemographic and labour market history variables) prior to estimating the program impacts. We applied the kernel matching technique, which uses the entire control group and re-weights control group members each time they are compared with a new participant, based on propensity score differentials. We used Inverse Probability Weighting (IPW) and Nearest Neighbour matching as alternative methods to validate the results. Note that these methods were combined with Difference-in-Differences (DID) estimation of program impacts. The DID method allows for the participants and comparison groups to differ on time-invariant unobserved characteristics, by assuming common time trends in the pre- and post-participation period in these characteristics. Figure 1 and 2 provide illustrations of matching and DID method.

Text description of Figure 1

This figure illustrates a simple example of matching method. A non-experimental propensity-score based weighting approach is used to measure the program impacts. This approach aims to ensure that the participants and comparison group are balanced in terms of background factors (that is, sociodemographic and labour market history variables) prior to estimating the program impacts. We applied the kernel matching technique, which uses the entire control group and re-weights control group members each time they are compared with a new participant, based on propensity-score differentials.

Text description of Figure 2

This figure illustrates a simple example of difference-in-difference method. Suppose, employment insurance claimants who participated in the program earn on average $8,000 more than they did at the start of the program, compared to an increase on average of $5,000 earned by similar individuals in the program. The net impact of the program would be an increase of $3,000 on average.

A brief description of propensity score model and difference-in-differences method is provided below.

Propensity score model

This report uses the logit regression model to estimate propensity scores. The propensity score is the conditional probability of participating in the program given the pre-participation variables. Propensity score matching uses the distance between estimated propensity scores to find similar individuals. The participants are pooled with potential comparison cases, and logistic models are estimated to predict the likelihood of participating in the OF program based on the background characteristics of the participants and the comparison group members. The data used for the model cover a large number of characteristics reflecting the labour market experiences and socio-demographic characteristics of participants and comparison cases. These characteristics include age, gender, marital status, and disability status. For both groups, the data have information on their economic region and province and qualifications (for example, occupational group, skill levels related to the last job before opening their EI claim, industry codes). Finally, the data also includes labour market history (including EI benefits, the incidence of employment, and employment earnings) in the 5 years preceding participation.

Difference-in-Differences (DID)

DID is one of the most popular methods to assess the causal effect on key outcome indicators of participating in an intervention or program. The DID method is used when outcome variables for participants and the comparison group are observed for 2 or more time periods (that is, before and after an intervention). The DID method requires that, in the absence of the intervention, the expected difference in outcome indicators between the participants and the comparison group is constant over time. This refers to a common trend assumption or constant bias assumption (Lechner, 2010). It is conventionally measured by comparing the observable trends in the pre-intervention period.

In order to use propensity score matching combined with difference-in-differences methods, this report applies following 3 identifying assumptions.

Conditional independence assumption

The conditional independence assumption requires that the common variables that affect participation assignment and intervention-specific outcomes be observable. An advantage of this analysis is that the available administrative data contains rich information about the type of disability for both participants and the comparison group. This allow for the inclusion of the most relevant variables influencing the decision to participate in the interventions and the labour market outcomes in the propensity score model. In this context, this report assumes that the Conditional Independence Assumption is satisfied.

Common support (overlap) assumption

This assumption ensures that persons with the same covariate values have a positive probability of being both participants and non-participants (Heckman, LaLonde, and Smith, 1999). In other words, each participant has one or more “counterparts” in the comparison group with the same covariate profile, such that appropriate counterfactuals can be constructed. The common support assumption can be verified empirically. The most straightforward way to do this is through a visual inspection of the density distribution of the propensity score in both groups. Figure 3 presents the distribution of the propensity scores before and after the matching for overall OF program. After matching, we observe a large overlap between the participants and comparison cases, and therefore a correspondingly high capacity to find suitable matches. A visual inspection indicates that there are no issues with common support for each intervention type (Figure C1 to Figure C4 in appendix).

Text description of Figure 3

This figure demonstrates a visual inspection of the density distribution of the propensity score in participants and control group before and after the matching for overall OF program. The left panel denoted as ‘Raw’ represents density distribution before matching and the right panel denoted as ‘Matched’ represents density distribution after matching. After matching, we observe a large overlap between the participants and control group, and therefore a correspondingly high capacity to find suitable matches.

Conditional bias stability assumption

The motivation for the conditional bias stability assumption comes from the concern that some relatively stable unobserved characteristics, such as ability, motivation, and/or attractiveness, may persistently affect labour market outcomes, but not fully capture conditioning on the available pre-program data. To satisfy this property, this analysis includes the pre-participation variables in the propensity score model.

This report uses the kernel matching algorithm to match participants and the comparison group with respect to their propensity scores. Kernel matching is a non-parametric technique that uses weighted averages of the outcomes of all individuals in the comparison group to construct the counterfactual. A major advantage of this approach is that it reduces the variance of the estimated effects.

Quality of the matching

The richness of the available administrative data allows for the inclusion of the most relevant variables influencing the decision to participate in the interventions and the labour market outcomes in the propensity score model. The use of socio-economic characteristics, pre-participation outcomes, transitions between different labour market states, regional information and labour market history decreases the likelihood of biased estimators of the true participation effects. This report uses a rich set of variables (Set of variables used for net impact analysis is listed below), including socio-demographic variables, labour market history of individuals (including EI benefits, the incidence of employment, and employment earnings in the 5 years preceding program entry). Previous empirical studies (Lechner & Wunsch, 2013) have emphasized the importance of these variables in influencing participation decisions and outcomes.

Set of variables used for net impact analysis

Socio-demographics characteristics

1. Gender
2. Age (at Action Plan Equivalent start date)
3. Disability Type (at the Action Plan Equivalent start date)
4. Self-declared status as a member of a visible minority
5. Self-declared status as a member of an Indigenous community
6. Marital status

Labour market characteristics

1. Occupation
2. Skill Level^{Footnote 7} and Type
3. Reason for Separation
4. Previous program participation up to 3 years preceding the participation
5. Social Assistance benefits received in each year prior to the Action Plan Equivalent start date (up to 5 years before starting the program)
6. Total reported earnings in each year before the Action Plan Equivalent start date (up to 5 years before starting the program)
7. Social Assistance income in each year before the start of the Action Plan Equivalent period (up to 5 years before starting the program)
8. Employment Insurance benefits received in each year before the start of the Action Plan Equivalent period (up to 5 years before starting the program)
9. Spousal income (up to 5 years before starting the program)
10. Average number of weeks of EI benefits used in the 5 years before starting the program

To illustrate the quality of the matching, Table 1 presents the summary results of covariate balancing tests before and after matching for each program intervention by using standardized mean differences between participants and non-participants. Propensity score matching substantially reduces the differences in the distribution of the covariates between participants and comparison groups. For example, for OF-SFE intervention the average cross-group standardized difference prior to matching was about 5.9, which decreases to 1.9 after matching. The reduction of differences in the covariate distributions between the participation and comparison groups is substantial, lending confidence to the results of the subsequent incremental impact analysis. A visual analysis of the density distribution of the propensity score in both groups (Caliendo and Kopeinig, 2005) was used to determine the common support region between participants and comparison cases. Further details on the full list of covariates used in the propensity score model, as well as the balancing scores, are available upon request.

Table 1: Mean standardized bias and pseudo R²

Program	# Participants before matching	# Participants after Matching*	Pseudo R2 before matching	Pseudo R2 after matching	Mean Bias before matching	Mean Bias after matching
Overall-OF	8,866	8,804	0.184	0.024	6.1	2.7
OF-EEAS	5,609	5,567	0.219	0.017	6.5	2.1
OF-SFE	1,144	1,135	0.103	0.008	5.9	1.9
OF-WS	705	701	0.068	0.004	5.7	1.4
OF-WE	853	850	0.11	0.007	6	2

• Source: Labour Market Program Data Platform, Employment and Social Development Canada.
• Note: *Number of OF participants remaining ( that is, after removing the unmatched participants) after the common support restriction is applied. Standardised bias is defined as the difference of sample means in participant group and matched comparison case subsamples as a percentage of the square root of the average of sample variances in both group.

After assessing the matching quality, the net impact estimates were produced using the propensity score matching combined with the difference-in-differences method. The results revealed that the estimated effects were not sensitive to different matching algorithms. The DID method allows for the participants and comparison groups to differ on time-invariant unobserved characteristics, by assuming common time trends in the pre- and post-participation period in these characteristics. The difference-in-differences method removes biases from unmeasured fixed confounder variables that affect both the participant and comparison groups over time. The validity of the difference-in-differences method depends on the stability of the bias between participants and non-participants in the pre-program period. This requires a sufficiently long pre-program period to properly control for the similarity between participants and non-participants, and to identify and deal with the Ashenfelter dip^{Footnote 8}. The current report uses a period of 5 years to observe pre-program behaviour. It also carries out a rigorous sensitivity analysis by applying Nearest Neighbor^{Footnote 9} and Inverse Probability Weighting matching^{Footnote 10} estimators. The findings from the sensitivity analysis suggests that the estimated effects are not sensitive to different matching estimators.

A potential limitation of this report is the possibility that there might have been pre-existing differences between the participants and comparison cases that were not measured during the matching process. These differences could have had an influence on the observed outcomes. For example, factors such as ability, health and motivation to seek employment were not directly measured except to the extent that they were captured in prior income and labour market attachment patterns. Although there is not data available for these variables, the pre-existing differences between participants and non-participants are well captured by the informative data on their respective labour market history (EI usage pattern and earnings’ profile) in the 5 years before participation and skills level related to their last occupation. Furthermore, the sensitivity analysis revealed that omitting unobserved characteristics in the propensity score model did not influences the sign of the estimated impact.

4. Socio-demographic profile of the participants

In this section, we discuss the sociodemographic profile of the OF participants overall. For details about the profile of participants by intervention types, please refer to table A1 in the appendix.

Most of the OF participants (n=5,567 or approximatively 63%) participated in the OF-EEAS intervention and approximately 13% of participants (n=1,135) were associated with interventions delivered under the OF-SFE. As shown in appendix A Table A1, participants were more likely to be male (52%) than female (46%). This trend applies to other OF intervention types except for SFE, which attracted a higher proportion of females (48% and 53% respectively) than males (44% for both). Overall, with respect to the type of disability, participants with mental health issues constituted the majority (24%), followed by those with physical disabilities (19%), developmental disabilities (13%), and learning disabilities (8%). The majority of participants were between 31 and 54 years of age constituting 51%, followed by those under 30 years (30%) and those 55 years or older (11%). The majority of the participants were non-Indigenous, non-visible minority and non-immigrant. The majority of the OF participants were single (65%), and resided in Ontario (40%) and British Columbia (15%). In the year preceding OF participation 58% of participants had no employment earnings, 54% received SA, and close to 0% received EI benefits.

5. Net impact analysis results

This section presents the results of the net impact analysis of the OF program on labour market outcomes. First, we discuss the overall results followed by detailed results for each intervention.

6. Cost-benefit analysis

The net impact analysis assesses program effectiveness using yearly impact estimates for the 5 year period after an OF program intervention (from 2011 and 2012 up to 2017). A robust cost-benefit analysis was conducted to estimate the costs and benefits of the OF program for the 10 year period post participation (5 years observed plus 5 years projected).

Cost-benefit analysis compares costs and benefits from the perspective of the individual (that is, the participant), the government, and when combined, the ultimate impact on society. Following the prevailing approach in the evaluation literature (see Heckman et al. (1999)), the net social benefit was measured by subtracting the costs of the program from their discounted social benefit streams. The cost-benefit analysis is based on quantifiable costs and benefits. The benefits from the individual and government perspectives are the following:

• employment earnings
• fringe benefits
• federal and provincial income taxes
• federal and provincial sales taxes
• employment insurance
• social assistance
• Canada Pension Plan (CPP) and Quebec Pension Plan (QPP) contribution
• employment insurance premiums

On the cost side, this analysis takes into account the direct costs of the interventions, which is the program costs. In addition, the indirect costs of the program are the marginal social cost of public funds^{Footnote 12} (MSCPF; see for example Browning, 1987) and foregone earnings. To account for the MSCPF, the direct resource costs should be multiplied by a factor greater than 1, in order to capture the distortions arising from financing the program by raising tax revenue. Following the advice received from expert peer reviewers, the MSCPF is calculated as 20% of the program cost and subsequent impacts on sales taxes, income taxes as well as on EI and social assistance outlays.

Cost-benefit analysis estimates the following 4 indicators.

Net Present Value (NPV): Following the dominant approach in the literature [Heckman, al., (1997), Oddbjørn et al., (2002), and Skipper et al., (2007)], the analysis determines the NPV 5 years after the end of participation by subtracting the program cost from the discounted benefits.

Benefits-cost ratio: It shows whether the amount of benefit generated from the program outweigh the costs incurred in delivering the program.

Payback period: It estimates the amount of time required for benefits to equal costs.

Social return: Social rate of return is the ratio between the net benefit to society and the direct program costs to the government. In other words, it shows the dollar value of the net benefit to the society when the government spends $1 on a program.

The main objective of this analysis is to assess whether the economic benefits outweigh the government’s investment in the OF program. Specifically, it aims to address the following questions:

• Did the program result in net financial benefits or losses 10 years after program participation?
• When does investment in the program “break even” (where the benefits equal the costs) from government and societal perspectives?
• What is the rate of return on investment of an intervention for the society after 10 years?

This analysis examines the benefits and the costs of the OF program from 3 perspectives.

• Individual: This perspective accounts for the costs incurred by participants when taking part in the intervention (foregone earnings) and the benefits accrued by participants following the intervention (higher-earnings profile relative to non-participants)
• Government: This perspective accounts for: the direct costs of the intervention borne by governments, as well as the costs associated with tax distortion; and, the benefits associated with participants’ higher earnings profile in the form of additional tax revenues and lower spending on income support programs
• Society: This perspective combines the individual and government perspectives

An important aspect of the cost-benefit analysis is to determine who bears a particular cost or benefit. What is a benefit from one perspective is a cost from another perspective. For example, a decrease in EI use is a reduction in income for the participants but is a benefit from the government’s perspective. Table 3 and 4 provide the costs and benefits from the perspective of the individual, government, and society.

The perspective from society combines both individual and government perspectives. For a given factor, a net gain to society occurs only when a gain to one group was not at the expense of another group. For example, increases in earnings represent a benefit for participants but neither a benefit nor a cost to the government. Thus, the net result is a gain for society. A cost to society occurs when a factor is a cost from one perspective and not a gain from the other perspective. For example, program costs represent a cost to the government, but not to participants; thus, they are considered a cost to society. Factors that constitute a net gain from one perspective but a net loss from the other perspective result in a zero impact from a societal perspective. For example, a reduction in EI benefits may represent a cost to participants and a benefit to the government and neither a cost nor a benefit from a societal perspective.

Figure 4 illustrates the costs and benefits of the overall OF program from the perpective of the government and the participant. The total net benefit for the participant is $7,803 and the total net cost for the government is $888 over a 10-year, post participation period. This translates into a social net benefit (combining government and participant perspectives) of $6,915 or a social rate of return^{Footnote 13} of 170% (9.44% per annum^{Footnote 14}).

Text description of Figure 4

Net present value for government

Components analysis	Government
Total Cost ($)	-$ 4,415
Employment Insurance ($)	-$ 87
Social Assistance ($)	$ 319
Government Deductions ($)	$ 3,295
Net present value ($)	-$ 888

Net present value for participant

Components analysis	Participant
Social Assistance ($)	-$ 319
Government Deductions ($)	-$ 3,295
Employment Insurance ($)	$ 87
Employment Earnings ($)	$ 11,330
Net present value ($)	$ 7,803

• Source: Labour Market Program Data Platform of Employment and Social Development Canada.
• Notes:
  1. Total cost includes the direct program cost and the loss (indirect cost) incurred by society when raising additional revenues (defined as Marginal Social Cost of Public Fund which is 20% of the direct program cost), such as taxes to fund government spending.
• 2. Government Deductions include EI premium, CPP/QPP contributions and income and sales tax
• 3. Employment Earnings include fringe benefits (for esample, employer paid health insurance, pension contributions) and earnings lost during time spent in the program.

Overall, $1 of investment in the OF program yields a return of $1.7 over the 10-year period following the intervention. From a societal perspective, the discounted benefits of the OF program exceeds the program costs in 3.8 years. All interventions had a positive social rate of return (Figure 5) and Skills for Employment intervention yielded the highest social return ($4.4) among all types of interventions.

Text description of Figure 5

Social return among different intervention types	Overall	Enhanced Employment Assistance Services	Skills for Employment	Targeted Work Subsidy	Work Experience
Social Return ($)	$ 1.70	$ 0.47	$ 4.40	$ 2.23	$ 0.81

• Source: Labour Market Program Data Platform of Employment and Social Development Canada.
• Note: ^{Footnote 1}The break even point occurs when the amount of investment made by the government generates an equal amount of benefits for society.

The cost-benefit analysis accounted for all quantifiable costs and benefits directly attributable to the program using the available data. The cost-benefit analysis is based on major quantifiable costs and benefits available in the administrative database. It does not include intangible benefits such as higher well-being for individuals related to better health outcomes.

In addition to producing improvements in labour market indicators, it is likely that active labour market programming such as the Opportunities Fund also has positive influences on additional social outcomes such as health. According to research evidence, labour market participation and higher income is associated with better health outcomes (for example, Canadian Institute for Health Information, 2015; Saunders & al. 2017). Therefore, expanding the cost-benefit framework to include potential savings resulting from improved health outcomes (in other words, reduced health system expenditures) could yield additional positive impacts for both individuals and the government beyond what is directly measured.

7 . Conclusion

The Opportunities Fund program led to a substantial gain in employment earnings, increased labor market attachment and reduced dependence on social assistance income of participants compared to the comparison group. The positive impacts observed are consistent across intervention types. The net impact analysis and cost-benefit analysis of the OF program generated robust evaluation results, deomonstrating that the program improves labour market attachment and yields a positive social return on investment.

The main estimation method used to determine the net impacts was propensity score kernel matching combined with the difference-in-differences technique. The econometric model was based on rich information about individuals’ labour market experience, including socio-demographic characteristics as well as labour market history. Other factors, such as severity of disability or education could also affect both participation and outcomes. However, these factors are not captured in the available data, and therefore may induce some bias in the results to the extent that they may not be equally distributed among participant and comparison groups.

Cost-benefit analysis shows that over the 10-year post participation period, the Opportunities Fund program yielded a significant net positive return on investment. From a societal perspective, benefits outweigh their associated costs in less than 4 years. Overall, $1 of investment in the Opportunities Fund program yields $1.7 in return over a 10-year period following the intervention.

The findings of the net impact analysis of the Opportunities Fund are consistent with the literature and previous evaluation studies of labour market programs employing similar techniques and methodologies. The Opportunities Fund facilitates the transition of persons with disabilities into the labour market, with associated positive long-term impacts.

References

CALIENDO, M., AND S. KOPEINIG (2005): “Some Practical Guidance for the Implementation of Propensity Score Matching,” Discussion Paper No. 1588, IZ

FREDRIKSSON, A. AND OLIVEIRA, G.M.D. (2019), "Impact evaluation using Difference-in-Differences", RAUSP Management Journal, Vol. 54 No. 4, pp. 519-532. https://doi.org/10.1108/RAUSP-05-2019-0112

EDGAR K. BROWNING (1987), “COMPARING MONOPOLY AND COMPETITION: THE INCREASING‐COST CASE,” Western Economic Association International, Volume 25, Issue 3, pp. 535-542.

HECKMAN, J. J. AND J. A. SMITH (1999): “The Pre-Program Earnings Dip and the Determinants of Participation in a Social Program: Implications for Simple Program Evaluation Strategies,” Economic Journal, 109(457), 313–348.

HERNÁN MA, ROBINS JM (2020). “Causal Inference: What If. Boca Raton: Chapman & Hall/CRC,” July 31, 2020

HÖFLER, M. (2005), “Causal inference based on counterfactuals,” BMC Med Res Methodol 5, 28. https:// doi.org/10.1186/1471-2288-5-28

KLUVE, J. Youth labor market interventions. IZA World of Labor 2014: 106 doi: 10.15185/izawol.106

KLUVE, J, PUERTO, S et al. 2016, Interventions to improve the labour market outcomes of youth: a systematic review of training, entrepreneurship promotion, employment services and subsidized employment interventions, Campbell systematic reviews, ILO, Geneva, viewed 27 Feb 2020, http://www.ilo.org/employment/Whatwedo/Publications/WCMS_508938/lang--en/index.htm .

LECHNER, M. (2010) "The Estimation of Causal Effects by Difference-in-Difference Methods," University of St. Gallen Department of Economics working paper series 2010 2010-28, Department of Economics, University of St. Gallen, revised Oct 2011.

LECHNER, M. and C. WUNSCH (2011) “Sensitivity of Matching-Based Program Evaluations of the Availability of Control Variables,” CESifo Working Paper No. 3381.

LEUNG, K. (2004): “Special Issue on Psychology of Severe Acute Respiratory Syndrome,” Asian Journal of Social Physiology, Volume 7, Issue 1, pp. 1-1.

LOCHNE, L. (2004): “Education, work and crime: Theory and evidence,” International Economic Review 45, 811-843.

ROSENBAUM, P. R. AND D. B. RUBIN (1983): “The Central Role of the Propensity Score in Observational Studies for Causal Effects,” Biometrika, 70(1), 41–50.

RYAN, A.M., KONTOPANTELIS, E., LINDEN, A. AND BURGESS, J.F. JR (2018): “Now trending: Coping with Non-Parallel Trends in Difference-in-Differences Analysis,” Statistical Methods in Medical Research, 3697-3711.

WING, C., SIMON, K. AND RICARDO A. BELLO-GOMEZ (2018): “Designing Difference in Difference Studies: Best Practices for Public Health Policy Research,” Annual Review of Public Health, 39(1), 453-469.

Appendix A: Detailed results on socio-demographic characteristics

• Source: Labour Market Program Data Platform of Employment and Social Development Canada.

• Source: Labour Market Program Data Platform of Employment and Social Development Canada.

• Source: Labour Market Program Data Platform of Employment and Social Development Canada.
• Note: *Disability Type was categorized into 5 broad categories from the original 11. This was done mainly to align with the data in the comparison group file.
• **Physical includes hearing, speaking, motor skills, agility, and physical.

• Source: Labour Market Program Data Platform of Employment and Social Development Canada.

• Source: Labour Market Program Data Platform of Employment and Social Development Canada.

• Source: Labour Market Program Data Platform of Employment and Social Development Canada.

• Source: Labour Market Program Data Platform of Employment and Social Development Canada.

• Source: Labour Market Program Data Platform of Employment and Social Development Canada.

Appendix B: Net impacts results

• Source: Labour Market Program Data Platform of Employment and Social Development Canada.
• Note: Significance level: ***1%, **5%, *10%

• Source: Labour Market Program Data Platform of Employment and Social Development Canada.
• Note: Significance level: ***1%, **5%, *10%

• Source: Labour Market Program Data Platform of Employment and Social Development Canada.
• Note: Significance level: ***1%, **5%, *10%

• Source: Labour Market Program Data Platform of Employment and Social Development Canada.
• Note: Significance level: ***1%, **5%, *10%

Appendix C: Distribution of propensity score before and after matching

Text description of Figure C1

This figure demonstrates a visual inspection of the density distribution of the propensity score in participants and control group before and after the matching for OF-EEAS program. The left panel denoted as ‘Raw’ represents density distribution before matching and the right panel denoted as ‘Matched’ represents density distribution after matching. After matching, we observe a large overlap between the participants and control group, and therefore a correspondingly high capacity to find suitable matches.

Text description of Figure C2

This figure demonstrates a visual inspection of the density distribution of the propensity score in participants and control group before and after the matching for OF-SEF program. The left panel denoted as ‘Raw’ represents density distribution before matching and the right panel denoted as ‘Matched’ represents density distribution after matching. After matching, we observe a large overlap between the participants and control group, and therefore a correspondingly high capacity to find suitable matches.

Text description of Figure C3

This figure demonstrates a visual inspection of the density distribution of the propensity score in participants and control group before and after the matching for OF-WS program. The left panel denoted as ‘Raw’ represents density distribution before matching and the right panel denoted as ‘Matched’ represents density distribution after matching. After matching, we observe a large overlap between the participants and control group, and therefore a correspondingly high capacity to find suitable matches.

Text description of Figure C4

This figure demonstrates a visual inspection of the density distribution of the propensity score in participants and control group before and after the matching for OF-WE program. The left panel denoted as ‘Raw’ represents density distribution before matching and the right panel denoted as ‘Matched’ represents density distribution after matching. After matching, we observe a large overlap between the participants and control group, and therefore a correspondingly high capacity to find suitable matches.