Guidance: Outcomes and measurement

Outcomes-based programming
(Re)Settlement Program Outcomes
Aligning activities with IRCC’s direct immediate outcomes
Ongoing performance assessment of direct immediate outcomes
Evaluating direct outcomes achievement
How IRCC measures direct outcomes at the program level
Aligning with IRCC indirect immediate outcomes
Ongoing performance assessment of indirect immediate outcomes
Evaluating indirect intermediate outcomes
How IRCC measures indirect outcome achievement

Outcomes-based programming

Outcomes-based programming is based on planning and managing a project with a focus on defining the desired client outcomes and measuring their achievement during the life of a project. For the purposes of the Settlement and Resettlement Programs, outcomes selected for a project should support the (re)settlement and integration of newcomers and align with IRCC’s (Re)Settlement Program outcomes.

Supported by inputs, including financial and other resources, the plans, activities and outputs of the projects should promote the achievement of the actual change in immediate (short-term) and intermediate (medium-term) outcomes that cumulatively lead to the achievement of ultimate (long-term) outcome(s). Outcomes should be measured to allow for the adjustment of project design, to address achievement challenges and/or to build on successful practices.

Figure 1 provides an overview of the outcomes structure and the main concepts. Service providers are encouraged to develop an outcomes-based approach in all funded projects. While outputs (i.e. client/service counts) are essential, outcomes should be what guides the project design and determines whether a project was successful.

Figure 1. From inputs to ultimate outcomes

(Re)Settlement Program Outcomes

The (Re)Settlement Program Logic Model (LM) captures the links between the activities/outputs, and the immediate, intermediate, and ultimate outcomes. In the LM, certain activities are expected to lead to a certain immediate outcome achievement. For example, language training activities should lead to an improved knowledge of an official language skills. From there, immediate outcomes can lead to one or more of the intermediate outcomes. For example, an increase in language skills should lead to a client being able to use an official language to function in Canadian society, or the ability for a client to connect to the Canadian labour market.

Your project must support the achievement of at least one of the IRCC (Re)Settlement Program’s immediate outcomes. Your approach to ensuring alignment should include three important elements of the outcomes story:

Aligning each activity in support of the associated IRCC immediate outcome;
Developing performance assessment approaches to assess immediate outcomes achievement throughout the project duration; and
Present the approaches for evaluation of the cumulative outcomes of the project.

Aligning activities with IRCC’s direct immediate outcomes

Through its (Re)Settlement Program, IRCC has defined a set of direct client outcomes. The immediate direct outcomes focus on the acquisition of skills or information and should take place any time between 0 to 6 months after the end of participation in an IRCC-funded service. A set of immediate outcomes of the program is provided in the Logic Model (more details on expected immediate outcomes are provided in Figures 3).

For each immediate outcome which your project supports, you will need to describe in the application:

How your project aligns with and supports the achievement of the relevant IRCC (Re)Settlement Program immediate outcome(s).
- If two or more of your activities align with the same outcome, clearly describe how they uniquely contribute to the achievement of that outcome. For instance, employment workshop and employment-relevant language training activities will both lead to achievement of the outcome Clients acquire knowledge, employment-relevant language skills, job search skills, connections and Canadian work experience, but each in a different way.
- Your project may support the full extent of the outcome or only an aspect of the given outcome, but the alignment must be clearly visible: For example, under the outcome Clients acquire knowledge, employment-relevant language skills, job search skills, connections and Canadian work experience, if you deliver activities to improve knowledge but do not improve job search skills, connections or Canadian work experience, describe that partial but explicit alignment with the IRCC outcome.
If relevant, how each activity within the project addresses the specific needs of the targeted client groups to facilitate the achievement of the outcome by that group.

Measuring performance

The next element of an outcomes-based approach is measuring the achievement of outcomes of your project. There are two parts to measuring performance:

Ongoing performance assessment of immediate outcome achievement, which is a crucial part of understanding whether a project is making a difference during its implementation and whether adjustments need to be made; and
A concerted effort to evaluate project performance at the mid-point and at the end of the project, including an assessment of intermediate outcomes achievement.

When developing your approach to measuring performance, you should reflect on the content of the Elements of Performance Assessment and Evaluation Framework in Figure 2.

Figure 2. Elements of performance measurement and evaluation framework

Ongoing performance assessment of direct immediate outcomes

When developing your ongoing performance assessment (see Figure 2 for the main components of a performance framework) include the following elements in your application:

Indicators that will be used to identify progress towards the outcome. If possible, define the targets for each indicator (i.e. a 15% improvement in participants’ knowledge compared to the starting point).
Methods and tools that will be used to acquire measurements of these indicators.
When and how often data will be collected.
How data will be analysed and how findings will be shared (i.e. reporting).
How analyses of these indicators will facilitate an ongoing improvement and management of the project and reporting on project achievements.

The description of the (Re)Settlement Program’s immediate outcomes and corresponding indicators is provided in Figure 3 and is meant to facilitate the outcomes-based approach to designing your project performance assessment strategy. You may wish to use similar or identical outcomes and indicators in your strategy.

Evaluating direct outcomes achievement

As per the (Re)Settlement Program Logic Model design, any of the immediate outcomes can potentially contribute to any of the intermediate outcomes, therefore it is not necessary to specify alignment with intermediate outcomes in the application. It is important, however, to reflect on which IRCC’s intermediate outcomes are relevant to your project while proposing the evaluation strategy. You should include a short summary of an evaluation plan that includes relevant outcomes, indicators, methodologies, and demonstrate the ability to organize activities to evaluate project results. You should also include how you will:

Reach clients in the longer timeframe to measure behavioural changes related to intermediate outcomes; and
Partner with evaluation or research organizations or identify sufficient internal knowledge in conducting intermediate outcomes assessment.

The descriptions of the (Re)Settlement Program’s intermediate outcomes and corresponding indicators are provided in Figure 4, and are meant to facilitate the outcomes-based approach to designing your project evaluation. You may wish to use similar or identical outcomes and indicators in describing your evaluation strategy.

How IRCC measures direct outcomes at the program level

For each of its expected immediate and intermediate direct outcomes, IRCC has developed a set of indicators. Please consult Figure 3 for more details on direct immediate outcomes and their indicators and Figure 4 for direct intermediate outcomes and their indicators. The main IRCC data sources for these indicators are the Immigration Contribution Agreement Reporting Environment (iCARE) and the annual Newcomer Outcomes Survey (NOS).

Outputs collected in iCARE are essential pieces of the outcomes measurement puzzle. Each organization delivering direct services is required to provide all output data (i.e. which clients received which services) in iCARE. This data is combined with outcomes information collected in the NOS to assess the overall outcome achievement of IRCC funded (Re)Settlement programming. When designing your reporting structure, include a plan for gathering and reporting information into iCARE on a monthly basis.
When developing your performance assessment and evaluation approaches you may wish to use the same or similar indicators shown in figures 3 and 4. You may use a survey design which asks similar questions to the NOS. This will allow you to understand if your project is making a difference vis-a-vis the outcomes reported by IRCC in its own assessments and evaluations. In your strategies, you may also include other measurement indicators that are relevant to your project, your services, and your clients.

Figure 3. IRCC direct immediate outcomes and performance indicators

Immediate outcomes	Description	Indicators
1. Access to IRCC-funded services is facilitated	IRCC aims to remove barriers to accessing settlement services by providing Support Services, such as childcare, interpretation and/or translation services, transportation, short-term counselling, provisions for clients with disabilities, and digital support in terms of equipment and client training in digital skills.	% of clients who identified a support service was needed and received that support service (ICARE) % of clients on a waitlist or indicating challenges to services access due to lack of support service(s) (iCARE/NOS) % of clients who indicated challenges getting services due to a lack of support services (NOS)
2. In-depth understanding of clients’ general and language needs and assets to facilitate settlement and language training pathways and outcome assessment	IRCC aims for all involved, IRCC, SPOs and clients themselves to understand clients’ needs and assets and to ensure that clients are referred to the most appropriate IRCC-funded or external supports through Needs and Asset Assessment and Referrals (NAARS) and Language Assessment. Those services also help develop baselines for measuring progress in addressing their needs.	% of clients who identified needs, by type (iCARE) % of clients who received a settlement plan (iCARE) % of clients receiving referrals/linkages by topic, including to Language Training (ICARE) % of clients assessed at each language level, by official language (iCARE)
3. High needs clients have their essential needs met and gain skills to facilitate their own settlement journey	IRCC aims to ensure high needs clients receive services and referrals that meet their identified needs. This outcome is linked to the Resettlement Assistance Program activities such as port of entry reception, provision of temporary housing, and creating linkages with other organizations. This outcome is also linked to delivery of intensive Case Management services and activities.	% of clients who received each of the RAP services (iCARE) % of clients who reported their immediate needs were met, for each need (NOS - RAP Only) % of CM clients receiving ongoing assessments (iCARE) % CM clients accessing other services during CM participation (iCARE)
4. Clients increase knowledge of life in Canada	IRCC aims to help clients by providing them with information about Canada on a variety of topics, such as educational opportunities in Canada, transportation, finance management, rights and responsibilities and knowledge about their community, including assisting French-speaking clients to learn about Francophone minority communities. Services help clients develop the knowledge they need to integrate through Information and Orientation services.	% of clients receiving information, by topic (iCARE) % of clients who indicated their knowledge of life in Canada changed as a result of IRCC-funded services, on variety of topics (NOS) % of clients who report usefulness of referrals, including non-IRCC, in meeting their needs (NOS)
5. Clients improve official language skills	IRCC aims to decrease barriers to integration due to a lack of official language knowledge by providing various types of language training. Improvements in literacy skills and general language knowledge are targeted through a variety of Language Training activities; improvements in conversation skills are also expected through Community Connection language learning activities.	% of language training clients who advanced to the next level (iCARE) % of literacy clients who achieve literacy levels (iCARE) % of clients (formal and informal) who indicate language learning gains for each skill, including literacy levels, in English or French (NOS)
6. Clients improve labour market knowledge, employment-related language skills, job search skills, connections, and acquire Canadian work experience	IRCC aims to increase clients’ knowledge about the Canadian labour market, conditions of employment, rights and responsibilities of employers and employees, job search skills, and improve connections to employment through Employment-relevant services and Employment-related language training.	% of clients, by service type (iCARE) % of clients who indicated changes in knowledge of the Canadian labour market (NOS) % of clients who improved connections to the labour market (NOS) % of clients who gained job experience (iCARE) % of clients who improved job-related language skills (NOS)
7. Clients improve connections to communities and social networks	IRCC aims to support newcomers to build connections where they settle, to develop a sense of belonging by increasing their knowledge and participation in the organizations within their community and to increase their networks with others. This develops the newcomers’ engagement and social capital, which are crucial for success in their new society. Community Connections activities play a role in supporting this outcome.	% of community-building services clients, by type and focus (iCARE) % of clients who indicate they have participated in at least one type of organization during a year (NOS) % of clients who indicate that they increased their social network as a result of participation in IRCC-funded services (NOS)
8. Newcomer students and their parents/guardians gain ability to function within the school and education system and increase settlement knowledge	IRCC aims to decrease systemic barriers to Canada’s education system for newcomer students and their parents/guardians by providing school-specific information to newcomers in Canada. Settlement Workers in Schools activities play a significant role in supporting this outcome.	% of clients who participate in SWIS-school oriented activities; by type of activity and topic % of clients who indicated change in knowledge on school related topics (NOS) % of clients who indicated change in knowledge about Canada (NOS)

Figure 4. Direct intermediate outcomes and indicators

Immediate outcomes	Description	Indicators
1. Clients access services that meet their needs	IRCC aims to ensure that clients receive appropriate needs assessments, referrals, and support services to facilitate access to services that support their settlement and integration journey. These enabling services should ensure a durable pathway that supports a client's access to services that address their settlement needs. This includes access to services in the client’s preferred official language as well as measuring alignment between needs and settlement services received.	% of clients who indicated the support services received met their needs (NOS) # and % of clients who received services in their official language of preference (iCARE) Alignment of identified needs, referrals and settlement services received (iCARE) # and % percentage of clients who indicated referrals received helped address their needs (NOS)
2. Clients independently access direct settlement services	Intensive and structured interventions aim to empower high needs clients to gain independence in their settlement journey. These clients should achieve similar outcomes to clients who did not receive enhanced supports.	#/% of clients who access direct settlement services after completing intensive RAP or CM services (iCARE) Settlement outcomes of CM clients similar (within 5% points) or better than those of clients who did not receive CM
3. Clients make informed decisions about life in Canada	As clients gain knowledge about Canada, they are empowered to make independent decisions in a variety of areas of their life, such as enrolling their children in schools, pursuing education, addressing emergencies, and/or conducting their own financial management activities. The changes are measured on each of the topics/tasks.	% of clients who indicate they are comfortable making independent informed decisions about life in Canada, by topic. Topics include emergency decisions, shopping, education opportunities, etc. (NOS) Changes of the indicated comfort level on the tasks and decisions with time (2-5 years in Canada) (NOS)
4. Clients use an official language to function in Canadian society	Teaching official languages aims to enable clients to fully participate in the labour market, society, and cultural life of Canada by improving a clients’ ability to use official languages in their everyday life, in their work environment, and in social situations. Topics for the indicators should be determined based on your project focus.	% of clients who indicate they are comfortable using an official language without help in daily life situations, by topic and official language (NOS) Talking with teachers Using language at work Using language in a social situation Changes in the indicated comfort level over time (2-5 years in Canada) (NOS)
5. Clients are connected to Canadian labour market	IRCC aims to facilitate newcomers’ participation in the labour market, to help clients contribute to the Canadian economy and achieve a good standard of living. Labour market participation may include working, looking for work as well as pursuing further education or other services to improve clients’ future employment opportunities.	# and % of clients by employment status (as indicated by client) (NOS) % of full time vs part time employment (NOS) Labour market participation rates: working and looking for work (NOS) Incidence of employment earnings for clients for years 1 through 5 since landing (NOS)
6. Clients participate in communities and institutions	The aim of this outcome is for clients to create social capital to help them cope with new challenges in their new society. It includes having a sense of belonging, including a sense of belonging to Francophone communities, and having reasonable trust in public institutions so that they feel they can turn toward them in case of need. Clients will create linkages with other newcomers and existing Canadians to increase their community connections.	# and % of clients who have a sense of belonging (to Canada, province, community, country of origin) (NOS) # and % of clients who indicate they have friends who can support them (NOS) # and % who volunteer (NOS) # and % of clients who indicate they have a high degree of trust in public institutions (NOS)

Aligning with IRCC indirect immediate outcomes

In addition to direct activities and their outcomes for clients, projects may also deliver activities to support the achievement of newcomer outcomes in an indirect way. IRCC’s LM specifies two indirect outcomes to reflect the purpose of the activities. The focus of indirect services is on further improvement of the functioning of the sector and its leadership and each organizations, increasing the capacity and tools and approaches to support delivery of direct services. Other activities support engagement of various non-settlement stakeholder groups and organizations involved in newcomer settlement and integration. Indirect services are an important part of the (Re)Settlement Program as they help to support a stronger sector, stronger services and a stronger society.

There are two indirect outcomes, each with a distinct focus and scope:

Projects that support the outcome
Settlement sector improves capacity to provide coordinated and effective settlement support to newcomers, including equity deserving groups, may have the following focus and scope:
- Direct service delivery projects with indirect capacity building components: In some direct service delivery projects a portion of funding may go towards indirect components for such activities as project-specific capacity building, tool development, or outreach and involvement of partners to accomplish a specific aspect of a project. For example, outreach to employers that support delivery of mentoring activities, or developing an intake tool to improve your client placement. The scope is to improve the ability of your organization.
- Fully indirect projects that focus on building the capacity of the entire sector, a network, or a group of stakeholders delivering a specific settlement service: This type of project can include the work of umbrella organizations or other representing bodies, to coordinate the work of all members, to create national approaches to service delivery, or to improve the ability of the settlement sector to apply GBA Plus strategies and/or to measure performance. The scope is to improve work of sector participants at regional or national levels.
Projects that support the outcome
Non-settlement partners improve capacity to provide support to diverse groups of newcomers across all sectors, in large, medium and small communities, including francophone minority communities may have the following focus and scope:
- Direct projects with indirect local non-settlement stakeholders: This includes projects with activities that influence the ability of local stakeholders to work with newcomers and address newcomers’ needs. For example, SWIS workers working with school staff, or a Case Management worker engaging with a local organization to address the needs of Case Management clients. For direct projects with indirect non-settlement stakeholders, the goal is to improve the ability of the local non-settlement organizations to address newcomer needs in a sustainable and ongoing way.
- Fully indirect projects that focus on a community or a specific stakeholder groups (employers): This type of project can include working with an entire community and multiple organizations in the community to build their capacity to address newcomer needs, to improve their welcoming community characteristics, and/or to improve the entire sector’s (e.g. employers) engagement and retention of newcomers.

In all four cases indirect activities should lead to one of the two indirect IRCC outcomes. It is important that the scope of your outcome (your organization/your local partners/the settlement sector or settlement service line/non-settlement stakeholders including employers and community institutions) and the focus for your outcome (improved capacity in GBA Plus, strengthened evaluation capacity, increased capacity to retain newcomers) is clearly indicated.

Ongoing performance assessment of indirect immediate outcomes

Regardless of whether your project is fully indirect or has an indirect component, the defined indirect outcomes also need to be measured at the immediate time frame. To do so you will need to develop approaches that will specify:

What indicators will be used to measure the outcome achievement of the immediate outcomes;
What tools will be used to collect information on outcomes of your project in terms of understanding the effects of the training provided, tools developed, meetings conducted, information shared on the knowledge and skills being addressed.
How frequently you will collect the data and from which group of stakeholders; and
How the analysis of the data will be used to inform project adjustments.

Evaluating indirect intermediate outcomes

While intermediate outcomes will not be identified automatically, intermediate IRCC outcomes relevant to your project should be included while describing the evaluation approaches in the Evaluation Section of the Application. In addition to inclusion of indicators, tools and methods for data collection, you should reflect on the following to support longer-term outcome measurements (e.g. ability to apply the gained knowledge or skills or improvement of community characteristics):

How the populations/community members/employers you trained or supported will be accessed to asses their change in applying the gained knowledge, skills and capacity changes;
Any engagement of evaluators or researchers that will facilitate assessment; and
How the findings will support reporting to your Board of directors or funders and further decision- making.

Concepts presented in Figure 2, Elements of Performance Measurement and Evaluation Framework, should be applied to the measurement of indirect outcome achievements.

How IRCC measures indirect outcome achievement

IRCC indirect indicators are shown in Figure 5 and may be used to support the development of indicators for your project. Most of the information is collected from SPOs though the Annual Report on Project Implementation and Outcomes (ARPIO) or from Local Immigration Partnerships (LIPs) though the Annual Performance Report for Community Partnerships (APRCP). IRCC also conducts ad hoc surveys on specific topics of interest with specific populations. Service Delivery Improvement (SDI) projects also provide an evidence base for informing the sector on innovative programming approaches. You may wish to use these listed indicators and/or add other indicators to assess the effectiveness of your specific targets, focus and partners.

Figure 5. Indirect outcomes and indicators

Outcomes	Description	Indicators
Immediate Indirect Outcome 9: Settlement sector improves capacity to provide coordinated and effective settlement support to newcomers, including equity deserving groups	This outcome aims to ensure that the settlement sector will continue to build its capacity to manage and deliver services in a coordinated and consistent way. This outcome supports sector-level coordination and representation as well as sector-level capacity building such as in applying GBA Plus to their organization and programing, promotion and advertisement of settlement services, service capacity improvements, development of service delivery tools, and the creation of innovation and evidence in support of more efficient and effective programming. This outcome is also supported at the organizational level when the organization focuses on building their own management or service delivery capacity (e.g. engaging local partners to support specific services such as indigenous local representatives providing information and orientation input).	Level of satisfaction of members/SPOs with coordination and representation of umbrellas (ad hoc survey) % of staff who received professional development training, by topic (ARPIO) Change in abilities of organizations to mange in a variety of areas and the effect on staff performance (ARPIO) Service delivery is consistent across the country (document review/ICARE) Satisfaction of SPOs with tools and supports available for service delivery (ad hoc survey) # of clients by demographic (iCARE) Evidence for best practices or validation of innovative approaches to settlement services (SDI reports, ARPIO)
Immediate Indirect Outcome 10: Non-settlement partners improve capacity to provide support to diverse groups of newcomers across all sectors, in large, medium and small communities, including francophone minority communities	This outcome is focused at the community level or specific engagement of non-settlement players to support newcomers in various sectors, including employers through employer engagement and capacity building, schools though SWIS, health providers through CM, and other stakeholders who play a role in the settlement and integration of newcomers. Through this engagement IRCC aims to improve a communities’ (mostly non-settlement stakeholders) ability to adjust their own services to be more inclusive and to meet the needs of their newcomer clients.	# and variety of non-settlement partners and their contribution to the project (APRCP, APRIO) Projects leveraging partners; additional resources and their effects (APRCP, ARPIO) # of employers supporting delivery of employment activities (ARPIO) Changes in perspective of partners on the needs, benefits and contribution of newcomers (APRCP)
Intermediate Indirect Outcome 11: Stronger sector, stronger services, stronger society	IRCC expects that the outcomes achieved at the immediate level will support more effective settlement and non-settlement supports, more inclusive and welcoming communities, improve Canadian labour markets’ ability to attract and retain newcomers, and improve public institutions’ ability to provide inclusive and relevant services to their newcomer clients. Overall there is an improvement in addressing the receptiveness of Canadian society to newcomers.	Evidence of changes by partners in their ability to work with and serve newcomers (APRCP, ad hoc surveys) Evidence of improved community acceptance of newcomers (APRCP, ad hoc surveys) Leveraging funds to support delivery of community activities to develop welcoming characteristics (APRCP) Evidence of improved attraction retention of newcomers by employers (ad hoc surveys, reports)

Page details

Date modified:: 2023-11-14

Language selection

Search

Guidance: Outcomes and measurement

On this page