Evaluation of the Social Development Partnerships Program

Introduction

The Social Development Partnerships Program (henceforth the "Program") is a grants and contributions program that supports the Government of Canada's priorities through investment in not-for-profit organizations, provincial and territorial entities (including educational institutions, health and social services institutions), municipalities and Indigenous organizations (including band councils, tribal councils and self-government entities). The aim of the Program is to improve the quality of life of persons with disabilities, children and families, Black Canadian communities and other vulnerable populations facing physical, economic and social pressures.^{Footnote 1}

The current evaluation of the Program covers the 2017 and 2018 to 2021 and 2022 period  and focuses specifically on the Disability component and Children and Families component.^{Footnote 2} This evaluation is completed in compliance with the Financial Administration Act and the Policy on Results. The Program's last evaluation was completed in March 2019 and covered the 2012 and 2013 to 2016 and 2017 period.

This is a process evaluation focusing on whether Program activities were aligned with the Government of Canada priorities and Employment and Social Development Canada (ESDC')'s objectives and been delivered as planned and does not specifically examine the efficacy of the Program. The evaluation questions, and the details of the lines of evidence used for this evaluation, including the methodology and their limitations, can be found in Annex C and Annex D.

Key Findings

1. The Program aligns with the Government of Canada priorities and ESDC departmental objectives on social development, including the priorities on diversity, inclusion and accessibility
2. Stakeholders and interviewed participants mentioned that additional and clear information (such as expectations, resources, eligibility, timelines, final reporting, etc.) may help clarify the application processes
3. Overall, the majority of funding recipients agreed that reporting requirements were reasonable and the Program's communication of these requirements was clear. However, evaluation found low uptake in the usage of the final reporting templates over the evaluation period. This finding should be considered within the context that the funding agreements do not mandate the use of customized templates
4. The Program's data requirements positively impacted the organizational capacity of the Disability component's funding recipients, while funding recipients from the Children and Families component experienced a neutral impact. Some ambiguities emerged regarding the Program's use of the collected data
5. All interviewed or surveyed stakeholders agreed that the Program increased social inclusion by decreasing barriers to participation, in Canadian society, while increasing opportunities for community engagement
6. Projects' approaches were generally sustained, beyond the funding terms, by funding recipients and partner organizations

Recommendations

The evaluation provides two recommendations.

1. The Program should explore different communications methods towards improved stakeholder access to information of the funding application intakes and how to apply
2. The Program should explore the barriers experienced by funding recipients related to reporting tools, to better support their performance assessment

Recommendation 1

The Program should explore different communications methods towards improved stakeholder access to information of the funding application intakes and how to apply.

Management response

Management acknowledges the need to efficiently and effectively communicate with stakeholders to ensure a seamless application process. The Program is already working to address this recommendation. For example, by offering different communications methods and considering ways to improve services in response to ongoing feedback. The introduction of the Program Design and Delivery Planning Framework has enhanced our approach to communications, ensuring a more effective and consistent dissemination of information about funding application intakes and application processes, in line with our commitment to continuous improvement and stakeholder engagement.

Recommendation 2

The Program should explore the barriers experienced by funding recipients related to reporting tools, to better support their performance assessment.

Management response

Management acknowledges the need to support applicants in completing reporting requirements. The evaluation found that the requirement to report was evident, however, the reporting expectation was not clearly outlined. As this evaluation period ended in 2021 and 2022, implementation of some activities and measures have already commenced and are ongoing. Funding recipients are required to collect relevant performance data and to report on the results of their projects. Ongoing communication aims to help ensure that these recipients are well informed of Program objectives, expected results and key priorities. This is expected to reduce some of the barriers they experience in reporting on results, improve the quality of results reported by them and enhance their projects' alignment with the Program's goals and priorities. With recent implementation of the Program Design and Delivery Planning Framework, we anticipate a more streamlined and effective approach in addressing the barriers faced by funding recipients. This will not only facilitate better performance assessment but also align the reporting processes more closely with the Program's evolving objectives and priorities, ensuring consistent progress and enhancement of our services.

Disability component

The Disability component provides funding for initiatives that help to improve the participation and inclusion of people with disabilities in all aspects of Canadian society. Funding, under this component, represents $11M and provides organizations with 2 types of funding.

1. Operational funding: supports initiatives that increase the organizational capacity of national disability organizations and regional disability organizations that support the social inclusion^{Footnote 5} of people with disabilities
2. Project funding: supports initiatives that address social issues and barriers that confront people with disabilities such as developing tools, resources and support services, and fostering partnerships and collaborative networks.

The evaluation of the Program's Disability component focused primarily on national operational funding, although funded projects were also included.

Children and Families component

The Children and Families component provides funding for projects that help improve the lives of children and families, and other vulnerable populations through increased social inclusion and participation. The focus of this evaluation will be on the grants and contributions project funding which accounts for $8.8M of that total budget.

The mechanisms and conditions under which the Social Development Partnerships Program and associated funding are delivered is often through an intermediary, or third-party, funding delivery model. This delivery model uses an approach where funding is provided to not-for-profit organizations with specialized knowledge of the needs and conditions in communities. In turn, these not-for-profit organizations allocate most of the funding to partner community and grassroots organizations who deliver projects at the community level.^{Footnote 6}

The Program addresses social development priorities by funding not-for-profit organizations' activities, approaches and operational funding that promote diversity, inclusion and accessibility.

Funded activities and operational funding supported the development and utilization of a range of approaches to address social development priorities such as diversity, inclusion and accessibility for vulnerable populations, as reflected in the examples provided in table 4.

The Program supported funding recipients through revising and adapting existing documents to better align with ongoing needs.

The Program updates and customizes its reference documents and tools to ensure ongoing alignment with the Government of Canada priorities and ESDC departmental objectives on social development including diversity, inclusion and accessibility, and continued address to existing and emerging social issues. Some of this work includes:

• updated the Terms and Conditions to improve performance measurement (such as additions of new outcomes) and address the needs of vulnerable populations due to the COVID-19 pandemic
• updated the Program's Performance Information Profile to include the separation and creation of the Program components' new Logic Models, and revised narratives and performance indicators
• created the applicant guide
• customized reporting templates for funding recipients for each component (as of 2019)
• the Disability component established a Gender Based Analysis plus (GBA+) template, in 2021, to help guide funding recipients' reporting and established a Performance and Accountablity Framework in collaboration with national disability organizations

The document review demonstrated that the Program maintains an extensive application process.^{Footnote 19} It includes:

• the launch of Calls for Proposals
  • information sessions:
  • supporting the Calls For Proposals by further explaining their requirements and answering questions
• a two-stage review process, including:
  • pre-screening of applications, regarding eligibility
  • assessment of the application's merit, including consideration of intersectional factors
• Ministerial funding decisions

Funding opportunities are thematic and vary in scope and objectives. Calls for Proposals outline, for example, its priorities, objectives and eligibility criteria. Additionally, the Program used 2 types of Calls for Proposals:

• competitive processes
• non-competitive processes

The competitive processes consist of:

• open Calls for Proposals
• targeted Calls for Proposals
• Expressions of Interest

The non-competitive processes included solicited and unsolicited processes.

However, insights into challenges or gaps, identified by key informants during the interviews, are included in tables 5 through 8. They highlight communication challenges such as gaps in awareness, clarity, access, and inclusion regarding the Program's application review process and the assessment of grants and contribution funding applications.

Please note, for tables 5 through 8, the word lack refers to a deficiency, not an absence. In addition, the word applicant refers to representatives from organizations that had submitted applications that had not yet received a funding decision or their application had not been approved.

The majority of funding recipients, from both components, indicated that the Program's communication of its reporting requirements was generally clear, however guidance on reporting expectations is needed.

An average of 57% of surveyed funding recipients from the Disability component and an average of 54% from the Children and Families component indicated that the Program's communication of its data collection and reporting requirements was clear.

The document review, and key informant interviews with Program officials and funding recipients noted that the data reporting requirement is referenced in funding agreements.

• More specifically, 48% of surveyed funding recipients from the Disability component, and 50% from the Children and Families component, reported that they found the Program's communication of its data collection requirements clear. While, 65% from the Disability component, and 57% from the Children and Families component found the Program's communication of the data reporting requirements clear (Figure 1).
• However, the majority of funding recipients, from both components, also indicated that they did not have a comprehensive understanding of specific expectations regarding the reporting requirements. Based on the document review, funding agreements are clear that reporting is required, but do not provide additional information on what is expected from funding recipients. Supporting evidence on why this disconnect exists is not available. However, it is noteworthy that the period of the evaluation includes years prior to when the customized reporting templates were created in 2019

Text description of Figure 1

Perceptions about communication	Disability component: data collection	Disability component: reporting requirements	Children and Families component: data collection	Children and Families component: reporting requirements
Accessible	48%	35%	43%	35%
Clear	48%	65%	50%	57%
Comprehensive	44%	26%	43%	36%
Relevant to needs	39%	35%	44%	21%
Culturally appropriate	4%	4%	21%	14%
Other	9%	9%	7%	7%

• (Source: Survey results from both Program components)

Please note that the 'other' category captured open ended responses and mostly reflected supplemental details to other categories.

There is low uptake in the usage of the Program's final reporting templates

Partly in response to recommendations from the previous evaluation^{Footnote 22} and stakeholder requests, in 2019, including a needs assessment conducted by the Disabilitycomponent, the Program began to develop customized final report templates, for each Call for Proposal.

The Disabilitycomponent's Program officials also reported that funding agreements reference that "data and information must be provided in a form acceptable to Canada" and do not require the use of templates. Accordingly, Program officials, from both components, noted that funding recipients were not required to use the customized reporting templates, as it was not stated as a mandatory requirement in the funding agreements.

However, completing final reporting templates are important for programs to ensure that responses to questions about program efficacy and efficiency are collected and reported. The completion of these templates allows the Program to obtain this information.

As such, given the aforementioned reasons the customized templates were created, the evaluation reviewed the administrative files to determine the frequency in which these templates were used, over the period of the evaluation. For the purposes of this review, both open (ongoing) and closed projects and operational funding agreements were examined. The results revealed a low uptake in the usage of the templates (Figure 2), showing that the majority (69%) of funding recipients from the Children and Families component, and a large majority (80%) from the Disability component did not use the Program's data collection and reporting tools.

Moreover, in examining the final reports submitted for both Program components, the review noted that when templates had been used, there was very little consistency in the way the required information had been reported and not all funding recipients answered all questions. Therefore, although the majority of funding recipients, fromboth components, reported that the communication of data reporting requirements was clear, based on the evidence found in the key informant interviews as well as the administrative data review, a comprehensive understanding of reporting expectations is not apparent.

Additional information gathered from the key informant interviews with funding recipients identified certain factors that may explain, in part, the low uptake in usage of the Program's final report templates. It suggests that there may be a connection between these results and identified challenges with the Program's communication about, and dissemination of its, tools on data collection and reporting; primarily related to the issue of lack of timeliness in providing funding recipients the final reporting templates and guidelines on reporting expectations (Table 9).

Text description of Figure 2

Funding stream where final reports were submitted	Children and Families component	Disability component
Contributions	40%	10%
Grants	29%	29%
Average	31%	20%

• (Source: Administrative data from both Program components)
• Note: These percentages were calculated by examining the final reports that had been uploaded into the Commons System for Grants and Contributions. The timeframe included the full period of the evaluation starting from 2017 and 2018, which includes years prior to, 2019, when the customized templates were created.

The analysis of the final reports comprised of a sample of eighty (80) reports. The Disability component comprised 49 final reports including 21 contribution agreements and 28 grant agreements. The Children and Families component comprised 31 final reports, including 12 contribution agreements and 19 grant agreements. Moreover, the extraction of the reports was done by evaluation during in the fall of 2022. However, additional final reports may have been submitted into the Common System for Grants and Contributions by the Program, after the evaluation's extraction period. Therefore, those final reports would not be included in the evaluation's analysis, including the percentages reflected in figure 2.

A number of data collection and reporting challenges were reported by the funding recipients.

• Notably, 30% from the Disability component and 44% from the Children and Families component stated that they did not receive any supporting tools from the Program
• Timely dissemination of data collection and reporting tools was perceived as one of the major challenges by the funding recipients (Figure 3)
  • Regarding the the Program's communication and dissemination of data collection requirements:
    • less than 40% of survey respondants from the Disability component considered the communication about the requirements to be accessible, easy to use, relevant and timely
    • less than 30% of survey respondants from the Children and Families component considered it to be accessible and easy to use, with less than 15% considering it to be relevant and timely
  • Regarding the the Program's communication and dissemination of data reporting requirements:
    • while the majority of survey respondants (52%) from the Disability component considered the communication and dissemination of data reporting requirements to be easy to use, less than 30% found it to be accessible, relevant and timely
    • 36% of survey respondants from the Children and Families component considered it to be accessible, easy to use and relevant, while only 7% considered it timely

Text description of Figure 3

Perceptions about communication and dissemination	Disability component: data collection	Disability component: reporting requirements	Children and Families component: data collection	Children and Families component: reporting requirements
Accessible	30%	39%	29%	36%
Easy to use	39%	52%	21%	36%
Relevant	39%	35%	14%	36%
Timely	26%	26%	14%	7%
Other	9%	13%	7%	14%

• (Source: Survey results from both Program components; Disability component (n=23), and Children and Families component (n=14))
• Please note that the 'other' category captured open ended responses and mostly reflected supplemental details to other categories.

The following table (Table 9) identifies some complementary, and supporting perspectives on the challenges related to the effectiveness of the Program's communication about its reporting requirements, and dissemination of associated tools. These were uncovered in the survey data and through key informant interviews.

• Note: The word "lack" references a deficiency, not an absence.

Program officials, who were interviewed from the Children and Families component, indicated they were not aware of whether the Program's data collection and reporting requirements increased funding recipients' organizational capacity to measure their intended results. However, the majority of Program officials from the Disability component, stated that it had.

The evaluation sought to gather information on the impact the Program's data collection and reporting requirements had on funding recipients' organizational capacity to measure their results (Figure 4 and Figure 5). This information was intended to provide a better understanding of whether the requirements helped funding recipients to manage their data collection (for example, setting baselines) and reporting (such as providing a structure for reporting). Improved organizational capacity to collect data and report on results could improve the quantity and quality of data provided in final reports; thereby increasing funding recipients' and the Program's ability to inform performance measurement. As demonstrated, in figures 4 and 5, the Program's data collection and reporting requirements impacted both components' funding recipients, with the Disability component showing primarily a positive impact on their organizational capacity and the Children and Families component showing primarily a neutral impact.

Text description of Figure 4

Disability component's perceptions about data collection and reporting requirements on organization capacity	Positive impact	No impact	Negative impact
Capacity to collect data	33%	28%	9%
Capacity to measure results	36%	24%	7%
Capacity for data management	33%	48%	4%
Capacity to report on results	50%	33%	4%

• (Source: Survey results from the Disability component. (n=23))

Text description of Figure 5

Children and Families component's perceptions about data collection and reporting requirements on organization capacity	Positive impact	No impact	Negative impact
Capacity to collect data	18%	39%	7%
Capacity to measure results	32%	36%	0%
Capacity for data management	21%	36%	7%
Capacity to report on results	32%	32%	0%

• (Source: Survey results from the Children and Families component. (n=14))

Diversity and intersectionality

The evaluation also explored whether the Program's requirements for data collection and reporting impacted funding recipients' ability to consider diversity and intersectionality.^{Footnote 23} The document review demonstrated that in 2021 the Disability component began to provide funding recipients with the GBA+ template and a link to a related information webinar to assist in reporting on diversity and intersectionality. The Children and Families component's reporting template includes questions related to groups served by the projects. The template allows for funding recipients to collect and report on information related to diversity and intersectionality.

Survey responses, from both components' funding recipients (n=40), reflected that the Program's data collection and reporting requirements had a positive impact on them. More specifically, 50% of funding recipients from Disability component reflected this experience. This was further supported with evidence from the administrative data review wherein final reports demonstrated common themes of increased knowledge and understanding of intersectional forms of discrimination and inequities affecting persons with disabilities. Moreover, 21% of funding recipients, from the Children and Families component, also reported that the Program's data collection and reporting requirements impact their consideration of diversity and intersectionality. Final reports, from this component, futher demonstrated examples of strengthened partnerships with organizations serving marginalized populations, needs assessments to better serve these populations, as well as innovative approaches to providing safe and inclusive spaces and materials.

Reporting requirements on funding recipients

To further explore impacts of the Program's data collection and reporting requirements on organizations, the evaluation also investigated whether they created a burden for funding recipients. While the majority (51%) of surveyed funding recipients, from both components (n=40), indicated that the reporting requirements were reasonable, they reported on the ways in which the requirements created a burden for their organizations (Table 10).

Additionally, interviews with Program officials, from both components, also reflected on funding recipients' organizational capacity to conduct data collection and reporting requirements. They indicated that funding recipients may lack capacity and resources to conduct this work and thus, may find the requirements burdensome. For example, it might be burdensome to a small organization, with limited staff, to prioritize data collection and reporting over project delivery such as serving clients.

• Note: The word "lack" references a deficiency, not an absence.

With regard to support provided by the Program to funding recipients, to assist with data collection and reporting, evidence was gathered.

• Interviews with Program officials, from the Disability component, stated that they engaged with funding recipients by providing guidance, as needed, follow-ups and monitoring of projects to enhance accountability and collect more data on results. Along with providing templates, some Program officials stated that they provided rationales on the need for data, and simplified requirements and terminology. Some funding recipients agreed that rationales were provided however, they received limited guidance on data collection and reporting
• Some interviewed funding recipients, from the Children and Families component, mentioned that the Program had provided limited guidance on data collection and reporting. Similarly, the majority of Program officials stated that they were unaware of any Program supports related to data requirements, although one Program official mentioned that they provided templates as well as guidance and direction when needed

There is a range of evidence on the usage of collected data.

The evaluation explored the Program's use of data collected from funding recipients. While certain aspects of data collection were supported by Program officials, it remains unclear how the data is specifically used to inform the Program's design or lessons learned.

• Some Program officials, interviewed from the Children and Families component, stated that the Program used data, collected and reported on by funding recipients, to inform lessons learned and Program design (such as the requirement of mandatory leveraging additional funds for projects was removed)
• The majority of Program officials, interviewed from both components, were unable to speak to how the data was used, nor were they aware of any data collection or reporting challenges. Some Program officials indicated that the use of data differs, depending on the Program area. Notably, varying roles and responsibilities within the Program and challenges related to internal communication, as previously highlighted, may hinder awareness of data findings and its usage

The Program is a partnerships program that funds many national organizations that are often situated in urban centres. Those organizations (that is, funding recipients) often partner with other organizations (in other words, partner organizations) across the country. The Program's provincial and territorial funding distribution, throughout the period of the evaluation, is presented in Annex F.

As defined in the Program's Performance Information Profile (2022), social inclusion refers to circumstances in which individuals can participate in, contribute to, and benefit from economic, social, political and cultural life which enhance the well-being of all people.

Both Program components share a similar ultimate outcome^{Footnote 24} and aim to increase social inclusion of their target populations.^{Footnote 25} All interviewed and surveyed stakeholders agreed that the Program has contributed to social inclusion through Program funding and its delivery model.

While all outcomes (intermediate and immediate) were evaluated, they were not always detailed separately for both components. Therefore, for the purposes of this evaluation, outcomes of both components were consolidated into four common outcomes. This approach reflects their similarities and seeks to focus on their main essence.

1. Increasing opportunities to access support for addressing social needs
2. Contributing to community engagement
3. Contributing to community participation
4. Improving abilities to address diverse barriers to social inclusion

The majority of funding recipients and partner organizations, from both Program components, reported that Program funding, to a large extent, was used to increase social inclusion of their target populations.

Funding recipients and partner organizations, from both components reported that Program funding, to a large extent, was used to increase social inclusion.

• Specifically, 89% of respondents (n=27) from the Children and Families component, and 93% of respondents (n=29) from the Disability component reflected these findings.

The COVID-19 pandemic affected the capacity of funding recipients and partner organizations from both components in achieving increased social inclusion.

While the pandemic raised challenges, funding recipients and partner organizations demonstrated adaptability by implementing innovative adjustments to their approaches towards achieving increased social inclusion for the Program's target population:

• the pandemic affected funding recipients and partner organizations to varying degrees (Figure 6 and Figure 7)
• interviewed funding recipients and Program officials, from both components, noted the crucial role of the Program's flexible support and stable funding in assisting organizations during the pandemic
• the survey data indicated that less than 37% of respondents in the Children and Families component and less than 30% from the Disability component reported negative impacts from the pandemic
• among the funding recipients who submitted final reports,^{Footnote 27} only 39% acknowledged the impact of the pandemic. They cited both positive and negative effects, likely due to factors including adaptability and Program support

Challenges and innovative adjustments

The challenges faced by the Children and Families component included the sudden loss of revenue due to the cancellation of fundraising events and logistical challenges linked to office closures, volunteer engagement, and staffing. The following are some of the adapted measures that were applied:

• enhancing mental health support services
• shifting to remote work
• improving technological skills
• using flexibilities, provided by Program officials, to use funding for unanticipated costs (such as larger spaces to accommodate physical distancing)

The main challenge faced by funding recipients and partner organizations from the Disability component was that their targeted population was at a higher risk of contracting COVID-19.^{Footnote 28} Some of the adapted measures that were applied included:

• fostering partnerships
• developing new approaches (for example, virtual events, re-designed best practices)
• increasing opportunities for social inclusion
• shifting to virtual communication and outdoor meetings

Text description of Figure 6

Intermediate outcomes	Significantly positive	Minimally positive	No impact	Minimally negative	Significantly negative
Increase opportunities for community participation	5%	11%	32%	21%	32%
Address barriers to social inclusion	11%	0%	26%	32%	32%
Contribute toward community engagement	21%	11%	11%	26%	32%
Increase opportunities to access support and address social needs	26%	11%	16%	11%	37%

• (Source: Survey results, Children and Families component (n=29))

Text description of Figure 7

intermediate outcomes	Significantly positive	Minimally positive	No impact	Minimally negative	Significantly negative
Increase opportunities for community participation	21%	4%	0%	13%	63%
Address barriers to social inclusion	13%	13%	0%	21%	54%
Contribute toward community engagement	21%	8%	0%	21%	50%
Increase opportunities to access support and address social needs	13%	13%	8%	21%	46%

• (Source: Survey results, Disability component (n=27))

Key finding #6. Projects' approaches were generally sustained beyond the funding terms, by funding recipients and partner organizations.

The Program's Performance Information Profile (2021) defined "project sustainability" as "the ability of a project to continue to implement some or all of its components such as activities, outputs, and outcomes beyond the funding term".

Recommendation 1

The Program should explore different communications methods towards improved stakeholder access to information of the funding application intakes and how to apply.

Stakeholders and interviewed participants mentioned that additional and clear information may help clarify the application and assessment processes Key Finding #2. Funding recipients, from both components, expressed concerns about the lack of awareness of the applications' two-stage assessment process, the long and onerous review process, and the technical issues experienced when completing the application. Program officials, from both components, raised the issue of a lack of internal alignment and effective communication between its policy and operations branches, along with difficulties with the short timelines to complete the applications. These challenges impact accessibility, as applicants are facing barriers in understanding and navigating the application process efficiently.

Recommendation 2

The Program should explore the barriers experienced by funding recipients related to reporting tools, to better support their performance assessment.

The evaluation findings reveal that the majority of funding recipients perceived the Program's reporting requirements as reasonable and communicated clearly. However, the evaluation found low uptake in the usage of the final reporting templates over the evaluation period, from both Program components. This finding should be considered within the context that the funding agreements do not mandate the use of customized templates. Moreover, in the cases when templates had been used, there was very little consistency in way the required information was reported. In addition, 30% of the survey respondents from the Disability component and 44% from the Children and Families component stated that they did not receive any tools from the Program. Additional challenges included timely dissemination of data collection and reporting tools was perceived as a major challenge by the funding recipients. Other challenges include, but are not limited to, lack of timeliness and clarity in communicating reporting expectations and limited guidelines on data collection and reporting Key Finding #3.

Document and literature review

The 16 document review sources were primarily derived from internal Program documents, while the 29 literature review sources were primarily derived from online search engines such as Google, Google Scholar, EBSCO, GCpedia, the Government of Canada Publications Catalogue and the Statistics Canada publications catalogue.

Key limitations

The Program's internal studies and documentation surrounding the current evaluation topics, was limited. Despite using the earlier stated search engines, access to and availability of extensive peer review articles were limited. Nonetheless, to mitigate these limitations, multiple lines of evidence have been included in the evaluation to complement evidence gaps in any particular line.

Administrative data review and analysis

Evaluation reviewed and analyzed administrative data from the Common System for Grants and Contributions (CSGC). Data for the period from April 2017 to March 2022 was submitted by funding recipients as part of quarterly progress reports and final reports' reporting requirements. Reports included information on activities, projected and achieved outcomes, as well as partnerships. The administrative data analysis of the final reports derived from a sample of 80 funding agreements (taken from the 130 unique projects or operational funding agreements files that were funded during the evaluation period) and was conducted using MAXQDA 2022.

The sample of 80 funded projects or operational funding agreement files was selected using a representative approach including consideration to both Program components, funding mechanism (grant or contribution), fiscal year covered in the evaluation period, geographical representation, and urban or rural representation.

Key limitations

There were also a number of limitations, which were specific to this line of evidence:

• data challenges related to data entry in CSGC as not all project files were complete, and it was unclear in some cases whether the status of the files were up to date
• not all components nor funding mechanisms were utilized each fiscal year (for example, there were no funded Children and Families grant projects funded in either 2020 and 2021 or 2021 and 2022)
• there was a low number of funding recipients located in rural areas. Final reports, and the amount of details contained therein varied widely regardless of whether the funding recipient used reporting templates
• of the 80 sampled projects, 9 projects (11% of sample) either did not submit final reports, or their final reports were not available at the time of the evaluation. Another 21 projects (26% of the sample) were ongoing at the time of report retrieval and consequently did not contain a final report. This represents 37.5% of projects in the sample. In particular, findings related to template usage were impacted by the unavailability of the final reports. Additionally, one of these files, within the Children and Families component, submitted two final reports for separate elements of their project, as opposed to submitting one final report, which may have marginally skewed frequency percentages as both were coded in the analysis of template usage

One area of focus for the Children and Families component was on projects related to sustainable development goals. These projects were funded under a separate funding source, from the rest of the component's projects. However, they were included in the sample of projects provided for the evaluation. As such, they were included in the administrative data review. Awareness of this limitation came during the final evaluation report stage. It was not possible to entirely remove this data. Nonetheless, related administrative findings, to the extent possible, were removed. The impact of this discrepancy was likely minimal as the evaluation results from the Disability component were mostly aligned with the Children and Families component.

Key informant interviews

A list of all funding recipients was provided by the Program, which was then analyzed by evaluators to remove duplicates. One-hundred and eighty-three (183) projects were funded within the evaluation period, and 130 were determined to be unique organizations. These 130 made up the purposive sampling frame for initiating the key informant interviews.

Key informant interviews with Program officials, funding recipients, partner organizations, applicants, and beneficiaries (also known as end-users) were conducted for the evaluation. The objective was to gain insight on the application process, use and impact of funding, as well as views on the extent to which the Program supported the achievement of outcomes.

The facilitation of interviews was done internally from June 8 to August 21, 2023. The interviews consisted open-ended questions. Five separate interview guides were created and used for both Program components' stakeholders. Interview questions were constructed internally and derived from the broader evaluation questions and associated indicators.

A total of 36 semi-structured interviews were conducted by online video, with a diversity of key stakeholders:

• ESDC Program policy officials from the Social Development Partnerships Program, including directors, managers and senior analysts from the Office of Disability Issues (n=2) and the Social Innovation and Community Development Directorate (n=2)
• ESDC Program officials from the Program Operations Branch, National Grants and Contribution Delivery Centre (n=4) and Program Oversight (n=2) including managers, senior analysts and senior program officers
• representatives of funding recipients (n=10)
• representatives of partner organizations (n=9)
• beneficiaries (also known as end-users) from the Disability component (n=9)
• applicants, including representatives of applications that had not yet received a funding decision or their application had not been approved (n =9)

The sampling of funding recipients, partner organizations, applicants as well as the Program officials reflected a purposive sampling approach. Specifically, for the funding recipients, partner organizations, and applicants represented diverse organizations, funding component, funding stream, fiscal year, geographical location as well as rural and urban profile.

Attempts were also made to have a purposive and representative sample of individuals who benefitied from funded projects (known as beneficiaries or end-users). However, the evaluation team was only successful in attaining contact information for beneficiaries from the Disability component. Consequently, the sample comprised of only 11 names, and thus a convenience sample resulted. Nonetheless, these participants represented various organizations, funding streams, genders, abilities, geographical locations, and roles and responsibilities.

Recruitment of beneficiaries, as mentioned, came exclusively through the Disability component. The beneficiaries had diverse abilities and barriers. Accordingly, some of the participants were able to respond to similar recruitment efforts that were used for other stakeholders, such as the email invitation that had been sent to funding recipients and partner organizations. They made up approximately half (5) of the 9 interviews and they each represented a project that had been funded by the Program. The remaining four interviews took place in two small group interviews, and those participants represented one specific funding recipient. The composition of these group interviews included youth (under the age of 30) who benefitted from the assistance of a learning support worker to participate in both the interviews as well as in the projects or operations of the organization, as funded by the Program. Learning support workers were present for both group interviews and their involvement was paid through Program funding.

Technical and subjective terms, such as third party delivery model and beneficiaries, were defined visually and orally, to limit the misinterpretation of questions. To preserve the integrity of interviews, interviewers refrained from leading participants' answers, asked for clarity when necessary and provided time at the end of each interviews for participants to comment or address aspects of the Program not raised.

Key limitations

Some funding recipients and partner organizations were not responsive or the email address provided by the Program was not accurate, resulting in bounce back emails. The evaluation team made several efforts to gather accurate information through internet searches and telephone calls. During the interviews, participants did not answer every question. Some questions or answers were clustered producing overarching responses. Additionally, some key informants may not have a clear recollection of their entire experience due to the passage of time since their involvement with the Program and when the interviews took place. Further, some terms may have been misunderstood due to subjective usage within the Program (that is, third party delivery model, or partnership model or intermediary model) and with respect to translation from one official language to the other (such as the terms funding recipients and beneficiaries). Moreover, the Program's Performance Measurement division was subsumed within the responsibilitieies of the Program's policy team in the spring of 2023, just prior to conducting the interviews with Program officials. This may have impacted some key informants' lack of awareness of reporting tools or data usage.

Written responses, to the interview questions, were encouraged for the following participants:

• organizations that had limited capacity to participate
• participants that did not respond to all interview questions but exceeded the permitted interview time and additional time provided (for example,1 hour and 30 minutes)
• participants that were unable to respond to all questions and were unable to continue passed the permitted time
• participants that were interested but could not, or had limited capacity to, participate during the interview facilitation timeframe

Two (2) written responses were submitted. They were reviewed by the evaluation team for quality assurance and, if required, participants were contacted to provide clarity on information submitted.

Lastly, as mentioned in the administrative data analysis limitations section, projects related to sustainable development goals, which were part of the Children and Families component, were included key informant interviews. However, only 2 of the 36 interviews were associated with these projects.

Online survey

A survey with Program funding recipients and partner organizations was conducted for the evaluation, to gain insight on aspects such as the level of satisfaction with the Program's data collection and reporting requirements, as well as views on the extent to which the Program supported the achievement of outcomes.

The survey was designed internally by the evaluation team and administered by ESDC's Internal Fact-Finding Service between July 12 and August 9, 2023. The questionnaire consisted of mostly closed-ended and multiple choice questions. A single survey was used for both funding recipients and partner organizations. However, skip logic was utilized throughout the survey to allow respondents to skip sections that were not relevant to their experience. The survey, and all associated communications, were offered in both official languages.

A list of all funding recipients was provided by the Program, which was then analyzed by the evaluators to remove duplicate organizations. One-hundred and eighty-three (183) projects were funded within the evaluation period, and 130 were determined to be unique organizations (in other words, funding recipients). It was these 130 that created the purposive sampling frame. From those funding recipients, contact information for 84 unique partner organizations was provided to the evaluation team. They were also included in the survey.

Each potential respondent was assigned a unique identifier and the survey was sent to 214 individuals (entire population) via email. Overall, the survey received 68 complete responses, for a response rate of 32%.

Limitations

There are several limitations, which are specific to this line of evidence. Program data entry was not always consistent as sometimes organizations appeared with multiple legal names, and not all information was entered. Notably, not all of the unique organizations had contact information. All partner organizations may not have had the opportunity to participate in the survey because it was only sent to organizations who were identified by a funding recipient, and not all funding recipients responded to the request for information about partner organizations. As the Program did not collect information on partner organizations, funding recipients received an email early in the evaluation process asking them to provide the names and contact information for their partner organizations. To avoid respondent duplicates, the evaluation team looked for duplicate postal codes in addition to legal names. To ensure that as many organizations as possible had an opportunity to participate in the evaluation, where contact information was not available, evaluators conducted online searches of organizations to gather information. In some cases the evaluation team called organizations to obtain a contact email address that could be used for correspondence. Additionally, some respondents may not have had a clear recollection of their entire experience due to the passage of time since their involvement with the Program and when the survey was conducted, and some terms used in the survey may have been misunderstood (for example, reporting requirements).

As mentioned above, for the Children and Families component, projects on sustainable development goals were funded under a separate funding source and should not have been included in the survey sample. Although, it was not possible to remove the data from the survey, its results only included 2 respresentatives from those projects. This represents only 3% of the survey data.

Use of Performance Information Profiles and Logic Models in program evaluation

The design and conduct of an evaluation heavily rely upon program Performance Information Profiles and Logic Models. In using these resources, for this evaluation, observations were made.

• the Program's Performance Information Profile was revised in 2021, and new indicators were included at that time. As such, baseline data and targets had not yet been determined^{Footnote 31} and thus could not be used as part of the evaluation. Moreover, no qualitative indicators were created for the Children and Families component. Nonetheless, the evaluation used the stated indicators, to guide the evaluation's methodology and provide reference for its lines of inquiry
• the Program components' Logic Models are very similar and contain many overlapping outcomes. Additionally, although language variations reflect some differences, the essence of the outcomes seemed synonymous. This created some challenges with collecting, disaggregating and analyzing Program data