Evaluation of the Immigrant Settlement and Adaptation Program (ISAP)

3. Evaluation methodology

The evaluation used multiple lines of evidence including both qualitative and quantitative methods. The following section describes these primary and secondary data sources as well as the strengths and limitations of the study. Data collection took place between July 2009 and February 2010.

3.1. Primary data sources

3.1.1. Interviews

Thirty-two interviews were conducted with three groups of ISAP stakeholders to collect information on all aspects of the evaluation. The groups of interviewees included CIC management (n=12), other CIC staff (n=15), and Provincial/Territorial representatives (n=5). The regional distribution of each group is shown in Table 3-1.

3.1.2. Surveys

In total, 814 survey questionnaires were completed by various ISAP stakeholders (Table 3-2). Four unique surveys were administered to various stakeholders including SPO managers, SPO staff delivering ISAP A projects, SPO staff delivering ISAP B projects, and clients.

3.1.3. Focus groups

Separate focus groups were undertaken with CIC officers, SPO representatives, and ISAP clients. In total, 18 focus groups involving 173 different ISAP participants/stakeholders were conducted in six cities across Canada. One focus group with SPOs was conducted in French. The purpose of the CIC and SPO staff focus groups was to collect data on Program need, delivery and management. Focus groups with clients on their experience participating in the Program. The breakdown of participants is shown in Table 3-3.

3.2. Secondary data sources

3.2.1. Document review

A detailed document review was undertaken with a particular emphasis on program relevance. The document review also provided information relating to particular issues of program performance, design and delivery. A wide variety of federal, departmental and program documents were reviewed as well as SPO materials including websites, training and promotional materials, program statistics, guidelines, tools and intake/needs assessment forms.

3.2.2. Literature review

The literature review included online and library materials and archives. The literature review provided contextual information regarding the need for ISAP, the relationship with other settlement programs, and a comparison of similar programs in other jurisdictions. A list of references can be found in Appendix B.

3.2.3. Administrative data analysis

The iCAMS database was reviewed and analyzed to prepare a profile of the service providers, program services, and clients. iCAMS is an Internet-based system through which SPOs provide CIC with information about their services and clients. iCAMS has collected information on ISAP A since April 2004^{Footnote 10}. ISAP B was not covered by iCAMS data as it does not include services provided directly to newcomers.

For ISAP, iCAMS collected information on the type and number of services provided as well as the number of clients receiving ISAP services. Clients reported in iCAMS are augmented by CIC’s Field Operations Support System (FOSS), which allows a comprehensive analysis of the profile of immigrants participating in the program as per their individual characteristics (e.g., gender, country of origin, age, etc.). SPOs can choose to use an individual or aggregate method of reporting on clients in iCAMS. The individual method of reporting is used by just over 80% of ISAP SPOs and allows linking clients and services. The remaining SPOs use aggregate or both aggregate and individual methods of reporting to iCAMS which prevents the linkage between clients served and services provided.^{Footnote 11}

3.3. Evaluation limitations and strengths

The methodology used in this evaluation had the following limitations:

• Lack of access to extensive reliable administrative data – Data on the numbers and characteristics of clients served and services provided is incomplete. To assess the representativeness of iCAMS data with respect to the number of reporting SPOs, the Integrated Financial and Material System known as SAP was used. SAP is a financial data system that tracks all funds committed by CIC and serves as a central repository of all contribution agreements (CAs) financial information^{Footnote 12}. A comparison of the data in iCAMS and SAP databases demonstrates that a significant percentage of SPOs are not reporting in iCAMS: according to SAP, between 2004/05 and 2007/08 contribution agreements were signed with 204 unique SPOs to deliver the ISAP A projects; however, only 142 of these SPOs (70%) reported data into iCAMS. Similarly, the SPO focus groups not only found that some SPOs are not reporting data in iCAMS, but also that some were unsure regarding whether they were doing so correctly. A review of the iCAMS data reported by each SPO showed that there are significant discrepancies in the number of services reported. Consequently, data for 31% (or 43/142) of reporting SPOs has been eliminated from the current analysis. As a result, the evaluation relies on information provided by 49% (99/204) of SPOs who received ISAP A funding. In addition, an operational constraint faced by SPOs is that some clients are unwilling or unable to provide their Permanent Resident Card number; thus, not all will be reported in iCAMS. All of the above constrained the use of iCAMS data to comment fully on the reach of the Program, to conduct year-to-year comparisons, and to conduct adequate cost-effectiveness analyses. As a result of these constraints, iCAMS data presented in this report should be considered with caution.
• Representativeness of data collected – As discussed previously, information on the entire client and volunteer populations was not available through the data reported in iCAMS. It was therefore neither possible to obtain a random sample nor to compare characteristics of the respondents to the surveys to that of their respective populations to determine if they were statistically representative. As a consequence, the results from the client and volunteer surveys can only be used as an indication of the perception of those two groups, and cannot be interpreted as being representative of the populations.
• Lack of information on client outcomes – While CIC collects financial information through SAP and output data through iCAMS, it does not have a systematic approach for collecting client outcome information. To support the new approach to settlement programming, CIC is currently working on addressing this shortcoming, although a new methodology for collecting data on outcomes will not be available for at least one more year. Thus, the evaluation relied significantly on surveys and focus groups to obtain information on client outcomes.

The evaluation used several methods to enrich the data collection and increase confidence in the overall results. The strengths of the evaluation methodology include:

• The use of multiple lines of evidence including qualitative and quantitative data allowed for the triangulation of findings;
• Multiple stakeholders were consulted, including four distinct groups of stakeholders (i.e., clients, SPOs, CIC representatives, and provincial/territorial representatives) which increased the reliability of data;
• Regional representation was obtained through the surveys, interviews and focus groups;
• The client survey was available in 9 different languages and offered through three different modes of communication (online, on paper, by phone) to increase the number and range of clients willing to participate; and
• Nearly one-third of the SPOs (32%) involved in delivering the ISAP Program participated in the evaluation.^{Footnote 13}

• Previous
• 1
• 2
• 3
• 4
• 5 (current)
• 6
• 7
• 8
• 9
• 10
• Next