Evaluation of the Citizenship Awareness Program

2. Methodology

2.1. Evaluation scope and approach

The evaluation approach and methodology were set out in an evaluation plan developed for both the Citizenship Awareness and Citizenship Acquisition, Confirmation and Revocation components of the Citizenship Program. The plan was developed between February and June 2012 and completed in consultation with all CIC Branches involved in their delivery. Although planning encompassed both components, each will be evaluated separately in order to ensure robust coverage of the different aspects of the program. A Terms of Reference for the Citizenship Awareness evaluation was approved by CIC's Departmental Evaluation Committee in July 2012.

The evaluation examines the Citizenship Awareness program activity (PA3.2.1 of the CIC Program ment Architecture (PAA)), including the Grant to the Institute for Canadian Citizenship (ICC). The Grant to the ICC was included in this evaluation as its objectives are ed with those of the Citizenship Awareness program. As well, combining the two elements into one evaluation was deemed to be an efficient means of clustering evaluations in order to meet requirements within existing resource levels.

The present evaluation focuses on the activities undertaken, outputs produced, and results achieved for Citizenship Awareness over the five-year time period between fiscal years 2007-08 and 2011-12. The formative evaluation of Citizenship Week conducted in 2004^{Footnote 9} solely on the activities related to Citizenship Week, and therefore provided little information to inform the present evaluation. An audit of the Citizenship Program^{Footnote 10} was conducted by CIC's Internal Audit Branch in 2011, and internal and external reviews were also conducted in 2010-11. However, these reviews primarily explored issues related to the Department's processing of citizenship applications, rather than the promotion of Canadian citizenship.

A program logic approach was used for the study design of the evaluation, supported by a detailed evaluation matrix. The logic model depicts the activities, expected outputs, and expected outcomes for both components the Citizenship Program (see Figure 2-1). The specific expected outcomes addressed in the evaluation of the Citizenship Awareness component are:

• Newcomers and established Canadians participate in CIC promotional events and/or receive CIC promotional materials;
• Clients of CIC promotional activities have knowledge of the responsibilities and privileges associated with Canadian citizenship;
• Eligible applicants become Canadian citizens and persons who are ineligible do not;
• Canadian citizenship is a valued status;
• Evidence-based recommendations and/or decisions are made; and
• Decision-makers and staff are equipped with the skills, knowledge and tools required.

Figure 2–1: CIC Citizenship Program logic model

2.2. Evaluation questions

The evaluation was designed to address two broad themes: relevance and performance. In keeping with the requirements of the Directive on the Evaluation Function (Treasury Board of Canada Secretariat, 2009), program relevance was assessed in terms of: (1) continued need; (2) consistency with respect to federal roles and responsibilities; and (3) ment with government and departmental objectives and priorities. Program performance was assessed by examining program results in terms of: (4) effectiveness; and (5) efficiency and economy (resource utilization). The evaluation questions, organized by evaluation issue, are presented in Table 2-1. The complete evaluation matrix, which includes specific indicators and methodologies for each evaluation question can be found in the Technical Appendices.

2.3. Data collection methods

The evaluation included the use of multiple lines of evidence to help ensure the strength of the information and data collected. Program representatives were consulted in order to obtain administrative data and program documents for review, as well as to assist in identifying potential interviewees and locations for site visits.

Following the completion of data collection, each line of evidence was analyzed and several brainstorming sessions were held with project team members to develop the overall evaluation findings, conclusions and recommendations. Each of the lines of evidence is described in more detail below.

2.3.1. Interviews

A total of 67 interviews were completed for the evaluation (Table 2-2). Interviews were undertaken with 5 key stakeholder groups. Individuals were selected in consultation with program representatives based on their knowledge of citizenship awareness activities. The interviews were conducted to respond to all of the questions in the evaluation matrix, covering areas of program relevance and performance.

Interviews were conducted either in-person or by telephone, and interviewees were provided with a copy of the relevant guide in advance of their interview (see the Technical Appendices for the interview guides).

Where qualitative information is presented in the report, the scale shown in Table 2-3 is used. Note that, in some cases (i.e., where the number of interviewees was too small or where the question yielded more descriptive information), the responses were not coded and a summary approach was used to analyze the information.

Table 2–3: Interview data analysis scale

All

Findings reflect the views and opinions of 100% of the interviewees.

Majority/Most

Findings reflect the views and opinions of at least 75% but less than 100% of interviewees.

Many

Findings reflect the views and opinions of at least 50% but less than 75% of interviewees.

Some

Findings reflect the views and opinions of at least 25% but less than 50% of interviewees.

A few

Findings reflect the views and opinions of at least two respondents but less than 25% of interviewees.

2.3.2. Site visits

Site visits were conducted in January and February 2013 in five communities: Vancouver, Calgary, Mississauga, Montreal and Halifax. The goal of the site visits was to observe the daily operational context of the program at the regional/local level and to better understand the interactions between citizenship processing and promotional activities. The experience also enhanced the evaluation team's understanding of the different perspectives of staff in the regional/local offices compared to those of staff in headquarters with respect to the overall relevance and impact of promotional activities. The following activities were undertaken during the site visits:

• Tours of the regional/local offices, including on-site testing and citizenship ceremony facilities;
• Interviews with CIC regional/local managers and staff, citizenship judges, and representatives of partner organizations (e.g., language training centres, settlement service providers and public institutions), to discuss their role in promoting citizenship;^{Footnote 12} and
• Observation of citizenship ceremonies, including an assessment against a set of standardized criteria related to facilitation and delivery. A total of nine citizenship ceremonies were observed: two in Vancouver, two in Calgary, two in Mississauga, two in Montreal and one in Halifax.

2.3.3. Ceremony exit surveys

Working with CIC regional/local office staff, a short exit survey was distributed to new citizen participants (18 years of age and older) at the ceremonies attended by evaluation team members during the site visits. The survey (and postage-paid return envelope) was included in their citizenship welcome package^{Footnote 13} and the ceremony clerk mentioned the survey in the ceremony instructions.^{Footnote 14}

The survey questionnaire included questions asking participants about the main reasons for their decision to become citizens, the parts of the ceremony that were most important for them, whether anything could have been improved, how the ceremony made them feel, and related impacts of their participation. See the Technical Appendices for additional information on the methodology as well the ceremony exit survey questionnaires.

A total of 217 new citizens completed the ceremony exit survey (59% completed it in English and 41% completed it in French). A slightly modified version of the ceremony exit survey was also created to be distributed at a few ICC ceremonies.^{Footnote 15} These ceremonies were not attended by members of the evaluation team. A total of 19 new citizens participating in an ICC ceremony completed the exit survey (all in English). The response rate, broken down by ceremony location, is illustrated in Table 2-4 below.

2.3.4. Survey of new citizens

A survey of new citizens was conducted to answer evaluation questions related to the design and delivery and performance of the program. The survey included questions related to the citizenship experience of new citizens (such as the reasons for their decision to become citizens and how important it was for them), as well as questions about the effectiveness of CIC's citizenship awareness activities and products (mainly the citizenship ceremony and the study guide).

A survey firm was engaged to conduct the survey by telephone with individuals 18 years of age and older who received their citizenship between 2007-08 and 2011-12, using a random sample of 10,000 new citizens, drawn from a population of 688,651 eligible clients in CIC's administrative databases.^{Footnote 16} In total, 657 new citizens completed the survey, mostly by telephone; 4% of respondents completed the survey online. With a confidence level of 95%, percentages reported for the survey of new citizens have a margin of error of ±3.8%. See the Technical Appendices for a more detailed description of the survey methodology as well as the survey questionnaire.

2.3.5. Administrative data review

The following types of administrative data were reviewed to obtain information on the operations of the program:

• Operational data from internal reports and the Global Case Management System (GCMS) were used to obtain information on the number of citizenship grants produced and the ceremonies delivered in the years under study;
• Administrative data from CIC communications function were used to identify the number, type, and cost of citizenship promotional activities conducted by the Communications Branch;
• Financial information gathered from SAP was analyzed to establish the costs for the program; and
• Web analytics data were used to assess program reach.

2.3.6. Literature review

A literature review was conducted to examine the evaluation questions related to program need and the role of the federal government. The research included academic and technical journals, publicly available information from foreign government websites, articles by think-tanks and/or non-governmental organizations (e.g. Organization for Economic Cooperation and Development (OECD)), as well as public opinion research. The review was also useful in gaining a better understanding of citizenship policy approaches and their influence on decisions related to citizenship promotional activities. The review also looked at approaches to citizenship promotion in other countries (Australia, New Zealand, United Kingdom, and the United States). See the Technical Appendices for a partial list of the articles and reports reviewed.

2.3.7. Document review

A review of over 50 relevant program documents was conducted to provide background and context to inform an assessment of the relevance and design and delivery of the program. Official government documents such as Speeches from the Throne and budget speeches, as well as policy and strategic documents were reviewed for contextual background and for information on CIC and GoC priorities. Legislation (e.g., the Citizenship Act) and regulations were reviewed to determine responsibility for citizenship promotion. Departmental reference documents including Department Performance Reports (DPR) and the Reports on Plans and Priorities (RPP), along with Ministerial speeches, news releases and internal program documents were used to address specific evaluation questions. See the Technical Appendices for a partial list of the documents reviewed.

2.3.8. Institute for Canadian Citizenship (ICC) case study

A special case study was conducted to examine the efforts and early results related to the ICC as part of the evaluation of the broader Citizenship Awareness Program, as well as to fulfill requirements under the Financial Administration Act for the grant to the ICC. The case study assessed the Grant to the ICC in relation to the objectives set out in the Grant Agreement. Findings from the case study have been integrated throughout the evaluation report, where appropriate, with a view to informing results for the program as a whole. The case study used four main methodologies in its examination of the activities and results of the ICC:

• Document review;
• Key informant interviews;
• A review of administrative data provided by the ICC; and
• Exit survey results from the ICC ceremony respondents.

A case study report was prepared (see Appendix A).

2.4. Strengths, limitations and considerations

The evaluation contained a balance of qualitative and quantitative lines of evidence and allowed for the triangulation of research findings. The strengths, limitations, their possible impacts on the analysis, and mitigation steps are discussed below.

The evaluation included numerous and diverse lines of evidence and solicited input from a large and varied set of stakeholders. This allowed many stakeholder perspectives to be included and also helps to ensure that findings are reliable. Findings documented by more than one line of evidence demonstrate consistency and thus increase the strength of conclusions.

In addition, site visits to the five different communities allowed for the first-hand observation of program operations in the regions, the delivery of citizenship ceremonies and the participation of new citizens in the citizenship ceremonies. It also allowed for the evaluation team to meet with CIC staff, judges and other stakeholders at the local level with a role in citizenship awareness and promotion, enhancing opportunities for formal and informal connections and the team's overall understanding of program implementation.

While mitigated by the use of multiple lines of evidence, some limitations should be noted:

• The evaluation focussed primarily on the perspectives of new citizens (and not permanent residents or those born in Canada) despite the fact that the target audience for the Citizenship Awareness Program is all Canadians. Understanding the perspectives of those permanent residents that do not choose to become citizens and those born in Canada would add depth to the findings; however, this was not feasible due to constraints on conducting public opinion research.
• A change in the Department's financial coding in 2011-12, and incomplete Citizenship Awareness expenditure information in the program budget, resulted in the use of estimates rather than actual figures in the analysis of resource utilization.
• Different internal reporting formats, as well as an operational change from tracking the number of on/off site ceremonies in favour of tracking the number of standard/enhanced ceremonies, made it difficult to determine the number of new citizenship grants for a particular ceremony type over the period of study.
• The response rate to the exit survey for those participating in ICC ceremonies was relatively low. Also, distribution of the exit survey was limited to ICC ceremonies held in the Greater Toronto Area (GTA) and Calgary. Therefore, results of the exit survey for those participating in the ICC ceremonies are not considered representative, and should be interpreted with some caution.

These limitations were taken into account when developing the findings, conclusions and recommendations put forward in this evaluation.

• Previous
• 1
• 2
• 3
• 4
• 5
• 6 (current)
• 7
• 8
• 9
• Next