Section 1: Evaluation of the Canadian Public Health Service Program at the Public Health Agency of Canada – Introduction

1. Introduction

1.1Purpose and scope

The purpose of this evaluation is to assess the relevance and performance of the Canadian Public Health Service program, situated in the Centre for Public Heath Capacity Development, at the Public Health Agency of Canada. The evaluation addresses requirements in the Treasury Board of Canada’s Policy on Evaluation (2009) for all programs to be evaluated on a five-year cycle, and was also requested by senior management to assist in program planning and decision making. This is the first evaluation of the Canadian Public Health Service program.

The evaluation examines the relevance of the program, including the continued need for the program and its alignment with federal government and Public Health Agency roles and priorities. It also assesses program performance including the progress made towards expected outcomes and the efficiency and economy of the program. The time period examined is between April 2006 and October 2012.

The following questions guided the data collection for this evaluation. All of these questions were covered by the evaluation. Key findings for each question were then synthesized into broader conclusions.

1.1.1 Relevance

• Continued Need:
  • What is the state of Canada’s workforce and capacity in public health?
  • Are the Public Health Agency’s Canadian Public Health Service activities responsive to Canada’s public health system’s needs? Are the goals and objectives of the Canadian Public Health Service program clear and well understood by all program stakeholders?
• Alignment with federal and Public Health Agency roles and responsibilities:
  • What is the federal role in building and maintaining Canada’s workforce and capacity in public health?
  • What are the links between the Public Health Agency’s Canadian Public Health Service activities and the federal role in building and maintaining Canada’s workforce and capacity in public health?
  • In what ways is the Public Health Agency best positioned to contribute to Canada’s public health system’s workforce and capacity needs? Does the Public Health Agency role complement or duplicate the roles of other stakeholders (other government departments, provinces and territories, local public health organizations, and non-governmental organizations)?
• Alignment with government and Public Health Agency priorities:
  • Do current Canadian Public Health Service activities align with federal government and Public Health Agency priorities? What are the gaps, if any?

1.1.2Performance

• Achievement of expected outcomes:
  • Has the program been implemented as planned?
  • Is the Canadian Public Health Service program achieving its expected outcomes? What has been accomplished through the Canadian Public Health Service activities?
  • What are the priorities and the vision for the Public Health Agency’s Canadian Public Health Service activities moving forward? How could existing activities be adjusted/enhanced to better address Canada’s public health system’s workforce and capacity needs?
• Demonstration of Efficiency and Economy
  • Is there variance between planned and actual expenditures? If so, what are the implications?
  • Are Canadian Public Health Service activities achieving their intended outcomes in the most economical manner? How have existing resources been leveraged to support the achievement of intended outcomes?
  • To what extent is there performance measurement in place? How is the program using performance measurement to improve the design and delivery of activities?

This report describes what the Canadian Public Health Service program does to help build and maintain Canada’s public health capacity. The evaluation uses the assessment of literature and documents, interviews, case studies and surveys. It assesses challenges and opportunities and presents key findings and conclusions. These, in turn, lead to three recommendations.

The scope of this evaluation does not include activities related to the Public Health Agency’s other field services programs, including the Canadian Field Epidemiology Program, the Laboratory Liaison Technical Officer program, the Office of Quarantine Services, and the HIV Field Surveillance program.

1.2Methodology

1.2.1Approach and design

This evaluation was conducted by the Evaluation Directorate, a group internal to the Public Health Agency but not involved in the program area responsible for the administration of the Canadian Public Health Service program.

This evaluation incorporates both qualitative and quantitative data to ensure a balanced analysis of the relevance and performance of the Canadian Public Health Service program.

1.2.2Methods

This evaluation used multiple lines of evidence, including a review of literature and relevant documents, as well as key informant interviews, surveys, and case studies. Performance measurement data, collected to report on activities associated with the Canadian Public Health Service program, was also used as a line of evidence for this evaluation. Lastly, an environmental scan of international programs related to public health capacity was undertaken.

Literature review

Evaluators conducted a review of literature to explore the magnitude of workforce and public health capacity in Canada. The purpose of this review was to obtain information about whether there was a need for the Public Health Agency’s activities in enhancing the workforce and public health capacity across Canada.

The articles reviewed included the literature (including unpublished literature) that program staff suggested, along with documents identified through Internet searches. Database searches were conducted in PubMed for English-only articles published after 2007. The search focused on public health need and capacity in Canada, by province, and internationally. A full listing of literature consulted is available in the bibliography.

File and document review

The Evaluation Directorate reviewed the files and documents available on the Canadian Public Health Service program at the Public Health Agency to explore issues related to program relevance and performance. Files and documents were initially identified by program staff and through a search of relevant websites. Lastly, the Public Health Agency’s website was searched for reports that mentioned the Canadian Public Health Service by name. In total, evaluators reviewed 79 documents, not including websites.

The types of documents reviewed included:

• program authorities and relevant legislation
• foundational documents
• strategic plans
• audits and other evaluation reports
• briefing notes and presentations
• performance measurement information
• financial data.

Interviews with key Public Health Agency senior managers and staff and key program stakeholders

Structured key informant interviews were conducted with former Public Health Officers, former site supervisors, current and former staff of the Office of Public Health, staff of Regional Operations, and senior managers at the Public Health Agency to explore issues related to program relevance and performance. In addition, interviews were conducted with senior provincial and territorial public health officials (predominately Chief Medical Officers of Health) to focus more specifically on federal-provincial-territorial public health workforce issues. Interviews were also conducted with sites that were unsuccessful in being matched with a Public Health Officer. Copies of the interview guides are available in the technical report.

Following six initial scoping interviews with key program contacts in late September and October 2012, additional interviewee names were suggested by program staff. From October to December 2012, 33 interviews were conducted, each lasting approximately one hour. The breakdown of interviewees by type of key informant is shown in Table 1.

Note: additional site supervisors, current and former Public Health Officers, and former students were interviewed via the case studies.

Surveys

The Evaluation Directorate conducted two on-line surveys with current Public Health Officers and site supervisors (who would not be interviewed via the case studies) to ask them about the relevance and performance of the program. The surveys employed a purposive sampling technique. After initial scoping interviews with key program contacts at the Office of Public Health Practice, contact information for current Public Health Officers and site supervisors was provided for the survey samples. Sample sizes and response rates were as follows:

• 11 Public Health Officers were invited to take part on a survey and 10 participated
• 16 site supervisors were contacted to take part in the survey and five participated.

Prior to the launch of the surveys, program staff sent a message to prospective respondents to apprise them of the evaluation and encourage their participation. The surveys were conducted simultaneously between November 27 and December 7, 2012. Fluid Surveys software was used to administer the survey and support data analyses. Copies of the questionnaires are available in a technical report.

Case Studies

During November and December 2012, three case studies were conducted to collect placement site level information about the implementation and impact of the Canadian Public Health Service program. Sites included one local organization each in Newfoundland, Nunavut and Ontario.

The following methods were used to collect data:

• review of internal documents and performance measurement data (e.g. site intake applications, Public Health Officer/placement site progress reviews conducted at 6, 12 and 18 months into placements)
• review of placement site documents pertaining to Public Health Officer activities (collected during site visits) (e.g. reports, publications)
• on-site observations
• semi-structured interviews conducted on site or by telephone (one former Public Health Officer), including: site coordinators (3), current Public Health Officers (3), former Public Health Officers (2), and a former student (1).

The choice of placement sites for the case studies was guided by recommendations from Public Health Agency program staff. Through a purposeful sample, three placement sites were invited to participate based on their geographic characteristics (one in the North, one in Eastern or Western Canada, and one in central Canada) and included a mix of newer and longer term placement sites. The sample of placements also represented the various backgrounds of the current and former Public Health Officers: an epidemiologist, a public health nurse, and a public health policy analyst. Site participation was voluntary; however, all of the sites approached agreed to be included in this study.

1.2.3Limitations

Most evaluations face constraints that may have implications for the validity and reliability of evaluation findings, conclusions and recommendations. This section illustrates the limitations in the design and methods for this particular evaluation. Also noted are the mitigation strategies put in place by the evaluation team to ensure that the evaluation findings can be used with confidence to guide program planning and decision making (Table 2).