Section 3: Evaluation of the Canadian Public Health Service Program at the Public Health Agency of Canada – Findings

 

3. Findings

This section provides a summary of the findings organized under two broad headings:

• Relevance: the need, federal public health role and priority in building and maintaining Canada’s public health capacity.
• Performance: the effectiveness, efficiency and economy of the Public Health Agency of Canada’s activities in this area.

3.1Relevance: Continued need, alignment with roles and priorities

This section provides details on the historical and ongoing nature of public health capacity gaps, the role of the Public Health Agency of Canada in addressing these gaps and how this issue relates to federal government and the Public Health Agency priorities.

As noted in the introduction of this report, Canada’s public health system’s capacity challenges were brought to the forefront in the landmark report issued by Dr. Naylor in response to the SARS crisis in 2003.^{Endnote 1} In this report, the specific public health system deficiencies that contributed to an inadequate response to the SARS outbreak were identified as a lack of surge capacity in the public health system and poor capacity for epidemiological outbreak investigation. In addition, although limited by poor quantitative data, the Naylor Report documented an inadequacy in the supply of public health professionals, inadequate public health training among practicing health professionals, and regional disparities as contributing factors to a reduced capacity to mount a public health response to SARS. In particular, the Naylor Report identified rural communities, Atlantic provinces, northern territories, and areas served by Health Canada’s First Nations and Inuit Health Branch, as geographic regions with shortages of public health physicians contributing to poor public health capacity. The importance of policy analysis in public health was also introduced by Naylor as a method to improve public health response.

More recently, reviews undertaken by the Public Health Agency suggest that these concerns are still relevant. In 2008, a review of the response to the listeriosis outbreak in Ontario noted a need to enhance outbreak epidemiological capacity and policy capacity.^{Endnote 10} In 2009, a report on the progress between SARS and H1N1 was released and identified the need to enhance public health capacity in provincial, territorial, and local public health organizations as an outstanding priority.^{Endnote 11} Lastly, a review of the 2009 H1N1 influenza pandemic revealed difficulty in the cross-government sharing of surveillance information, and a continued need to strengthen capacity to prepare for and respond to pandemic influenza.^{Endnote 12}

Currently, quantifying the public health workforce, and by extension its capacity, is challenging.^{Endnote 13} Although several attempts have been made, the Canadian public health workforce has not been successfully counted. Despite this, there currently exist some recognized gaps in the public health workforce, similar to those identified by Naylor in 2003. These include gaps related to specific professions (epidemiologists and policy analysts) and regional disparities (northern territories, Atlantic provinces).

Despite difficulty in defining and measuring public health capacity regionally, there is a common understanding in the literature and amongst key public health stakeholders that public health capacity in Canada varies by region. The documented high prevalence of public health issues (e.g. infant mortality, cancer mortality, and suicide / injuries that are amenable to public health intervention) in various regions across the country infers a greater need for enhanced public health capacity in these regions.Endnote 1 For example, the northern territories have high rates of tuberculosis, chlamydia, teen births, smoking and substance abuse, while the Atlantic provinces have high rates of non-communicable diseases.Endnote 1, Endnote 14 It is well documented that these regions of the country report difficulty in recruiting epidemiologists and senior public health officials with formal graduate public health training; however, this difficulty is also reported in larger provinces including Ontario.Endnote 15, Endnote 16

Similarly, as part of the data collection for this evaluation, interviews with senior provincial and territorial representatives identified the northern territories, provinces with smaller populations, and rural areas of larger provinces as having particular difficulty in the recruitment of appropriately trained public health professionals. In particular, a number of senior provincial / territorial representatives indicated that the recruitment and retention of epidemiologists is difficult. This finding is supported by the literature review which revealed that there is a recognized long-term need for epidemiology and its related infrastructure in Canada.^{Endnote 15}, ^{Endnote 17}, ^{Endnote 18} A review of Canada’s response to recent outbreaks noted that the lack of epidemiologists and related infrastructure has limited our response.^{Endnote 10}, ^{Endnote 12}, ^{Endnote 16}, ^{Endnote 19}

A review of the site applications submitted to the Canadian Public Health Service program during the first ever round of site solicitation provides further support to the finding that there is a need for specific public health professions across the country. Of the proposals submitted (n=66), approximately 48% of sites requested the assistance of an epidemiologist, while 23% of the requests were for policy analyst support.^{Endnote 8}

While less clearly documented, there also appears to be a need for nursing staff (in public health and clinical health care) both in Canada and internationally. This shortage is likely to be compounded in the future by the expected retirement of up to 1/3 of nurses by 2015.^{Endnote 20}  In addition, the relative age of community health nurses is older than other nursing groups, and is expected to be disproportionately affected by the impending retirements resulting not only in a loss of nurses but also their roles as mentors and teachers for future nurses.^{Endnote 20}, ^{Endnote 21} It is important to note the challenge in documenting this need due to the difficulty in discerning differences between community care, public health and clinical nursing.

The federal role

In Canada, public health is a shared responsibility between federal, provincial and territorial and municipal governments, the private sector, non-government organizations, health professionals and the public. While provinces and territories have the primary responsibility for their respective human resource and capacity needs, the consequences of their capacity gaps can affect the public health system as a whole. Response to public health events may be national in scope and beyond the capacity of any particular province or territory to address by itself. As such, the Government of Canada may choose to play a supporting role in this situation. Subject to governmental policy considerations, the spending powers may be used to create non-regulatory programs in areas of provincial jurisdiction.

Through legislation, the Department of Health Act establishes the Minister of Health’s powers, duties and functions relating to health including “cooperation with provincial authorities with a view to the coordination of efforts made or proposed for preserving and improving public health.”^{Endnote 22} Further, the Public Health Agency of Canada Act (2006) established the Public Health Agency and signaled its intent to cooperate in the field of public health with provinces and territorial governments.^{Endnote 23} The mandate of the Public Health Agency is to assist the Minister in exercising or performing the Minister's powers, duties and functions in relation to public health.

The Public Health Agency’s role in addressing Canada’s public health capacity needs generally aligns with and supports the Agency’s public health mission. The Public Health Agency’s mission is to promote and protect the health of all Canadians, through leadership, partnership, innovation and action on public health. The Public Health Agency’s role includes: intergovernmental collaboration on public health; the facilitation of national approaches to public health; emergency preparedness and response; and the prevention and control of chronic and infectious diseases. Through the Canadian Public Health Service activities, the Agency is performing this role and working towards this mission.

The involvement of the federal government in helping to address Canada’s public health capacity needs is supported by recommendations from the Naylor Report.^{Endnote 1} As noted by Naylor, the public health system’s collective ability to respond to public health events is limited by the weakest jurisdiction in the chain of the public health system. This led Naylor to conclude that there is a federal responsibility to strengthen public health system capacity. The Naylor Report went on to recommend that the federal government create a National Agency for Public Health that would have responsibility for the development of a national public health service. More specifically, the report recommended that this national public health service contribute to the development of public health professionals capable of supporting federal programs and augmenting provincial, territorial, and local public health programs to address system capacity gaps and limitations. As a direct result of these recommendations, the Public Health Agency was created in 2004. The Public Health Agency subsequently developed the Canadian Public Health Service program in 2006 to help address Canada’s public health capacity needs.

Since the Naylor Report, it has been recognized that there is a federal role in the development of workforce and capacity for public health through the “provision of scarce or technical resource and surge capacity” related to public health.^{Endnote 1} Naylor developed the initial program structure for the Canadian Public Health Service and describes the important federal role in building public health capacity by “strengthening individual skills and by sharing knowledge across jurisdictions” (Naylor Recommendation 7.3).^{Endnote 1} Although Dr. Naylor articulates the federal role the most clearly, several other pan-Canadian reports support a similar federal role in renewing public health by increasing the numbers of public health professionals and by the establishment of a program of secondments among Canadian jurisdictions with the goal of renewing public health in Canada.^{Endnote 24}, ^{Endnote 25}

The federal role is further defined in program authorities received to address Avian and Pandemic Influenza Preparedness. The Public Health Agency received funding to address public health capacity through activities to:

• build a skilled and responsive federal public health workforce across Canada
• provide the critical public health capacity required to prepare for and respond during a public health emergency.

The program authorities also outline that these activities will involve the deployment of public health staff to address federal as well as provincial/territorial and local priorities.

Provincial and territorial perceptions of federal role

Analysis of feedback collected from interviews with senior provincial and territorial representatives reveals that they view the Public Health Agency’s role with respect to Canada’s public health system’s capacity as one of coordination and support. More specifically, interviewees felt that the Public Health Agency is best able to contribute to the country’s public health capacity needs by facilitating and coordinating knowledge exchange across the country, especially during emergencies (surge capacity), and by increasing the training that they offer public health organizations (particularly to those with the greatest needs so as to strengthen the system on the whole). Interviewees also expressed a desire for the Public Health Agency to lead in the efforts to influence training and academic programs, through development of core competencies, standards and/or guidelines.  Interviewees were especially complimentary of the work previously done by the Agency on core competencies for public health, and were eager for additional work in this area to continue.

International comparison of the federal role in building public health capacity

Internationally, most countries are struggling with the concept of public health capacity. Although strategy and role varies by country, respective central bodies are consistently involved in enhancing public health capacity. Of the countries reviewed, only the Center for Disease Control in the United States includes programs to enhance public health capacity similar to the public health officer role and training to future public health professionals like the Canadian Public Health Service student component. Despite this, all countries attempt to influence the outcome of enhanced public health capacity but used activities such as training opportunities for public health professionals, grant funding to non-federal public health organizations, and informing the development of competency documents.

The Canadian Public Health Service does not appear to duplicate the roles of other stakeholders or programs

Evaluative evidence collected from the literature, document review and interviews with senior provincial and territorial representatives suggest that the Public Health Agency’s role in delivering the Canadian Public Health Service program does not appear to duplicate the roles of other stakeholders. As part of this evaluation, no evidence was identified that described similar programs or initiatives being offered outside of the Public Health Agency. However, within the Public Health Agency there are other ‘Field Services’ programs also aimed at strengthening public health capacity in Canada. These include: the Canadian Field Epidemiology Program, the HIV Field Surveillance program, the Office of Quarantine Services (now: Office of Border Health Services), and the Laboratory Liaison Technical Officer program. All of these programs are focussed on addressing specific elements related to public health capacity in Canada; however, each of them employs a unique approach to do so. An overview of the various field services program within the Public Health Agency is provided in Appendix B.

The field service program most similar to the Canadian Public Health Service program is the Canadian Field Epidemiology Program. The two programs share common elements, including the long term objective of strengthening public health capacity in Canada. However, while the Canadian Public Health Service program was envisioned as a mechanism to enhance public health capacity by providing public health partners across Canada with skilled professionals in a number of disciplines, the Canadian Field Epidemiology Program was established as a short-term (2-year) professional development program for public health epidemiologists, and is part of an international network of Field Epidemiology Training Programs. Its focus is on providing training and building skills and experience in applied epidemiology, while also providing service to public health partners. While Public Health Officers are indeterminate Public Health Agency employees, and rotate between Canadian Public Health Service placement sites on a 2-year cycle, Field Epidemiologists are hired or assigned for only a 2-year term. Finally, in the Canadian Public Health Service program, placement sites are evaluated for their need, and Public Health Officers are matched to the sites based on their skills and professional goals. Conversely, in the Canadian Field Epidemiology Program, epidemiologists are placed in jurisdictions that already have good capacity to provide mentoring, learning and projects which meet the program's professional experience guidelines.

While there does not appear to be any obvious duplication of activities, the Canadian Public Health Service program has the potential to overlap with the Canadian Field Epidemiology Program, as they both place emphasis on the training of epidemiologists.

At the time the Canadian Public Health Service program was created, the Federal Budget (2006) included specific funding for the Public Health Agency directed towards pandemic preparedness and emergency management readiness. This was followed with continued direction through the Speech from the Throne (2010) which stated that “protecting the health and safety of Canadians and their families is a priority of our Government.” Following this, the Federal Budget (2012) also committed to “…improve response capabilities to food-borne illness emergencies.” This underlines the level of priority that the Government of Canada places on Canada’s capacity to respond to public health demands.

The Public Health Agency’s activities are centered on promoting and protecting the health of Canadians through leadership, partnership, innovation and action in public health and directed to creating healthy Canadians and communities in a healthier world. As such, the Public Health Agency’s Report on Plans and Priorities (2006 to 2012), consistently highlights public health capacity as a priority.Endnote 26-31 Initially, strengthening workforce and enhancing public health capacity is noted (2006); however, the priority evolved to “facilitate Canada-wide coverage of public health professionals with the capacity to respond to public health events. This includes placing public health practitioners in northern sites and the recruitment and mobilization of field epidemiologists and public health officers across Canada.” Further, the 2011-12 Report on Plans and Priorities includes three specific plans to address this priority, namely:

• Strengthen the capacity of the public health workforce,
• Build public health capacity in the North, and
• Improve systematic knowledge sharing and use of best practices in public health.^{Endnote 31}

The 2011 Public Health Agency’s Corporate Risk Profile identifies public health workforce and capacity as one of its top ten public health risks, stating that there is a risk that “Canada lacks the appropriately trained workforce, tools, organization capability, and inter-jurisdictional systems to respond to and plan for public health threats.”

Additionally, the Public Health Agency’s Strategic Horizons document indicates four strategic directions. The goals of the Canadian Public Health Service program align with three of them:

• leadership on health promotion, disease prevention, and health protection which includes activities such as pandemic preparedness, outbreak management, and surge capacity;
• strengthened public health capacity and science leadership which includes activities such as building public health capacity through a focus on provincial and territorial public health needs; and
• enhanced emergency preparedness and response which includes the capacity to mobilize the health emergency surge capacity.^{Endnote 32}

3.2Performance

This section discusses how the design and delivery of the Canadian Public Health Service program have contributed towards achievement of intended results.  The progress the program has made towards achieving its intended results is also presented.

The complement of individuals that have served as Public Health Officers since program inception have reflected the profession-related public health capacity needs discussed in section 3.1.1. This includes epidemiologists, policy analysts and public health nurses. However, in terms of regional disparities, the locations of Public Health Officer placements to date do not appear to reflect those with the greatest capacity needs.

Between 2006 and April 2013, a total of 42 Public Health Officer placements occurred through the Canadian Public Health Service program. These 42 placements involved a total of 29 Public Health Officers (some of whom were placed multiple times) and took place in a total of 30 organizations in 20 communities across Canada (Figure 1). While some of these placement site locations reflect areas with known capacity needs (as discussed in section 3.1.1), others do not. For example, the northern region (all territories combined), which represents an area with known capacity needs, has only been the recipient of approximately 17% of all Public Health Officer placements. The eastern region (all Atlantic provinces combined), another area with known capacity needs has been the recipient of approximately 26% of Public Health Officer placements. Combined, these two regions, which represent the seven provinces and territories with the most commonly acknowledged public health capacity needs, have only received 43% of the Public Health Officers placements. To date, Public Health Officer placements have not occurred in the province of Quebec due to Chapter M-30 of the Quebec legislation, which states that “no public agency may, without the prior written authorization of the Minister, enter into any agreement with another government in Canada or one of its departments or government agencies, or with a federal public agency”.^{Endnote 33} This means that the majority (57%) of Public Health Officer placements have occurred in the remaining five provinces (Ontario, British Colombia, Alberta, Saskatchewan and Manitoba).

A more critical analysis of the Public Health Officer placements to date provides additional support to the notion that the Canadian Public Health Service program may not be addressing the areas with the most critical public health capacity needs. While Regional Coordinators have liaised with provincial/territorial and local stakeholders to identify their capacity needs/priorities via the solicitation of proposals, feedback from senior provincial/territorial representatives suggests that the program is still not reaching the priority areas of their provinces/territories. As expressed by senior provincial/territorial representatives during interviews, the specific areas/communities within each of their provinces/territories with the greatest capacity needs are the rural and remote areas. However, as shown in Figure 1, the majority of Public Health Officer placements to date have taken place in larger urban centres. This is not to suggest that public health organizations located with larger urban centres do not have capacity needs of their own (for example the First Nations Health Authority in BC), but rather, to emphasize that these are not the priority areas within the provinces and territories identified by senior provincial/territorial representatives. Senior provincial/territorial representatives suggested that this information may not be getting back to the Canadian Public Health Service program because individual sites are solicited directly, rather than coordinated through the provincial/territorial offices. Of note, several provincial/territorial representatives stated they were not aware that any Public Health Officers had ever been placed in their province (all provinces and territories have had at least one Public Health Officer).

Figure 1 – History of Public Health Officer placements across Canada (2006 to 2013)

The feedback from senior provincial/territorial representatives is supported by feedback collected from placements sites that applied successfully to the Canadian Public Health Service program, but were unsuccessfully matched to a Public Health Officer. A total of five unsuccessfully matched placement sites were contacted as part of this evaluation, all located within larger urban centres. In each case, interviewees indicated that in the absence of a Public Health Officer, they either found another way to address the project work described in their application for the Public Health Officer, or else they deemed that it was not a priority and it was thus not completed. Though the generalizability of this feedback is limited, as this group represented a small sample of the unsuccessfully matched sites from the 2012 site application process (total n = 23), the fact that these sites were often able to complete the project work in the absence of a Public Health Officer does raise questions about the extent of the capacity needs in these larger urban centres.

The fact that the majority of Public Health Officer placements have occurred in larger urban areas may be partly attributable to a number of issues. Firstly, the inclusion of Public Health Officer preference for placement location in the current matching process may work in favor of the larger urban centres. While Public Health Officers select their site preferences based on a number of factors, Public Health Officer feedback indicates that geography is an important consideration and that some regions may be less desirable to most Public Health Officers. Unfortunately, for some provinces/territories (e.g. Saskatchewan and Nunavut in the most recent round of rotation), despite having multiple projects included in the site catalogue, they may be unsuccessful in acquiring a Public Health Officer for this reason.

Secondly, after each round of site solicitation, there are typically fewer Public Health Officers available for rotation than there are successful site applicants. For example, in the 2012 rotation process, there were nine Public Health Officers available for rotation, but there were 38 successful site applicants. This resulted in 29 successful site applicants, which included rural and remote areas, not being matched with a Public Health Officer. While the program leaves these sites in the catalogue for an indefinite period of time (in hopes that they will be successfully matched in a subsequent rotation), there is no guarantee that these sites will receive a Public Health Officer. Of note, when possible (i.e. for sites located in the north), the program has made efforts to provide Canadian Public Health Service graduate students to unsuccessfully matched sites to help offset the lack of a Public Health Officer. It is unclear how often this student substitution has occurred or how effective this student substitution effort has been.

Thirdly, feedback from both successful and unsuccessful placement site applicants indicated that completing the application for a Public Health Officer can be a resource-intensive activity. Interviewees suggested that this may limit the number of sites that can apply to the program, as those with the greatest capacity needs may not have the staff or time resources to apply.

Thus, the Canadian Public Health Service program does not appear to be addressing the areas within Canada’s public health system with the greatest capacity needs and this may be partly attributable to the program’s poorly defined intended results and site selection criteria, as described in the next sections.

While intermediate and long term outcomes of the Canadian Public Health Service program are included in the recently updated logic model, the precise manner in which program activities are expected to lead to the achievement of these outcomes, and whether these activities are aligned with the original program intent, is less evident. This situation has contributed to a variety of perceived program priorities and outcomes among stakeholders. The program’s intended results are not well-defined, and the program’s design lacks a clear, consistent and commonly understood strategic approach.

Partially attributable to this lack of clarity is the limited availability of detailed program documents (strategic and descriptive). In addition, the authorities for the Canadian Public Health Service covered Avian and Pandemic activities more broadly, and therefore limited details were included with respect to the specific intent and rationale for this program. While the Canadian Public Health Service aims to strengthen public health capacity, the term ‘capacity’ has not been consistently articulated or defined by the program^{Footnote a}, nor does it appear to be universally interpreted by program stakeholders in the same manner. Further, while work is currently underway within the Agency to define and describe what is meant by public health capacity, at the time of data collection, there did not appear to be a commonly accepted Agency-level understanding of this terminology. A variety of interpretations of what constitutes public health capacity were offered by program stakeholders during the key informant interviews.

Similarly, a significant challenge for this evaluation was the lack of an explicit and agreed-upon program theory. The program does not have a management approved logic model summarizing the program theory. A logic model (Appendix A) was provided for the purpose of the evaluation; however it was incomplete as the two ultimate outcomes for this program were still under development at the time this evaluation was being conducted.Endnote 3 Program staff noted that the ultimate outcomes in the logic model were still under development because they were waiting to align them with the Branch-level expected results which were not yet completed. Further, the immediate and intermediate outcomes did not have clear linkages to activities and outputs, and the logic model lacked an accompanying narrative. Narratives could increase understanding of the overall logic of the program or elements within it.  The program recognized these gaps and made early attempts to identify activities and outputs as part of the evaluation process, and is currently developing a more substantive logic model.

The lack of well-defined program outcomes and unclear program theory has contributed to a variety of perceived program priorities and outcomes among stakeholders. In the scoping stage of this evaluation it became apparent that it was not clear to stakeholders how the Canadian Public Health Service program should work. There were a variety of opinions offered by key informants about the objectives of the program and how best to achieve them. The majority of program stakeholders (internal and external) seem to agree that the primary focus of the program should relate to addressing site need; however, while the majority of program management and staff feel that the program is currently doing this, senior provincial/territorial representatives feel that the program is currently placing the development of Public Health Officers as the priority. Similar inconsistencies were observed in survey and interview feedback collected from placement site supervisors. Additionally, survey results indicate that the majority of current Public Health Officers do not feel that the program objectives are clear and well understood by all program stakeholders.

The program’s site selection criteria (Figure 1, page 21), which guide the site selection process, reflect the lack of clarity in the program theory. Currently, the program assigns a weighting of 25% towards ‘demonstrated need’, and a combined weighting of 35% towards ‘opportunities for professional development’ (20%) and ‘type of supervision’ (15%).^{Endnote 7} This weighting distribution appears to misrepresent the program theory, given that these outcomes are equally weighted in the program’s logic model. Further, the criteria are not well defined and have undergone multiple iterations since program inception. For example, it is not clear what is specifically meant by the criterion ‘demonstrated need’, or how ‘potential impact’ is assessed.

In addition, the site selection criteria appear to be paradoxical. While the program’s intermediate level outcomes (addressing placement site needs and facilitating Public Health Officer learning/development) are reflected in the site selection criteria, the current program delivery model may not support the achievement of both. The paradox is that the sites with the greatest need may not have the resources to facilitate Public Health Officer learning and may consequently be unsuccessful in being matched with a Public Health Officer.

The site selection criteria also do not appear to reflect or support the activities outlined in the original program mandate (described in section 3.1.2). For example, although the strategic alignment of Public Health Officer projects with Public Health Agency priorities was envisioned as part of the original program mandate, the current site selection criteria (Figure 1, page 21) do not include a criterion reflecting this intent. Similarly, the original mandate also envisioned the Canadian Public Health Service program focussing on the planning and management of disease outbreaks, surveillance and emergencies (surge capacity); however this also appears to be missing from the current selection criteria.

The site selection criteria may also be missing links to current program priorities. For example the program’s recently updated long-term outcome includes a statement about strengthening the Public Health Agency presence in the North. This priority does not appear to be represented in the site selection criteria. As such, it is unclear what strategies are guiding site selection in support of, or alongside these criteria. For example, it is not clear if the program is aiming to place Public Health Officers in priority areas (like the North), or looking for equal representation across the country. Actual Public Health Officer placements to date do not provide any further indication, as neither historical nor current Public Health Officer placements demonstrate that the north has been a program priority (as discussed in section 3.2.1).

The lack of detail associated with these criteria was raised by site applicants (both successful and unsuccessful) as a challenge. Successful and unsuccessful site applicants revealed that it was not specifically clear what was meant by each of the criteria and they were not completely apprised of how the final decisions were made by the program when reviewing site applications. They indicated that this made it challenging for them to submit their most competitive application. Despite these design and delivery related challenges, the Canadian Public Health Service program has made some progress towards achieving its intended intermediate outcomes for placement sites, Public Health Officers and students. This progress is described in the next sections.

This section reviews the extent to which the Public Health Agency has achieved its intermediate outcomes related to enhancing the capacity of partner organizations. Progress towards this outcome has been made through both Public Health Officer and student contributions at placement sites.

Public Health Officers have contributed to placement sites through a variety of activities in both leadership and supportive roles.  Public Health Officer project work has spanned a range of public health topics that include: communicable, zoonotic, and non-communicable diseases; injuries; and environmental health impacts (Table 4).

Enhanced ability of placement sites to address their identified public health gaps

The Canadian Public Health Service program has made progress towards its stated intermediate outcome of helping to address public health gaps at participating placement sites. This has been facilitated through both Public Health Officer and student placement efforts.

Through both the survey and interviews, site supervisors expressed that the ability of their organizations to address workload capacity gaps was improved by the temporary addition of a Public Health Officer. This included an improved ability to address ongoing core public health work as well as self-identified priority projects that they were previously unable to pursue. Some site supervisors indicated that this additional resource has helped their organizations to become more proactive, permitting, for example, the development of work plans that include more ‘upstream’ activities such as epidemiology projects on issues of sexual health and environmental health risks, and training on emergency preparedness and response to local staff. Examples of work undertaken by Public Health Officers are noted below.

Additionally, some site supervisors also reported that their organization’s response to public health events was improved by the addition of the Public Health Officer. Specific examples of Public Health Officer contributions during local and national outbreaks were cited. For example, during the H1N1 outbreak, one site supervisor noted that their Public Health Officer contributed significantly to an epidemiological report on the first case of H1N1 in the country. Another example involved a Public Health Officer getting involved in the investigation and management of a syphilis outbreak in Nunavut.

This feedback from site supervisors is supported by the perceptions of current and former Public Health Officers, Regional Coordinators, and program managers, who all felt that the work of Public Health Officers has helped to address priority needs at placement sites and, to a lesser extent, alleviated the workload of essential staff.

Similarly, although the primary intent of the student placement is to fulfill educational requirements and build the competency of the student, the majority of site supervisors indicated that the student projects supported organizational capacity gaps by allowing them to undertake discrete projects that may not otherwise have been accomplished in the desired timeframe. It was noted that due to the short-term nature of a student placement (usually for the duration of one school term), the project scope and expectations of the supervisor, student, university and Public Health Agency staff need to be very clearly defined prior to the start of the placement. In addition, while students require the motivation and ability to work independently, the demands of a northern placement also require sufficient mentorship to make the placement a success.

Increased capacity / ability of placement site staff

Evaluation evidence indicates that the Canadian Public Health Service program has also made progress towards achieving its stated intermediate outcome of enhancing capacity of partner organizations to fulfill their public health needs. Site supervisors reported that Public Health Officers have contributed to their sites through the provision of technical skills such as epidemiological surveillance and analysis. In addition, site supervisors noted that Public Health Officers have also contributed to the development of tools, high-quality reports, as well as the delivery of training to site staff.

Similarly, Public Health Officers indicated that they have enhanced the capacity of placement sites through the provision of subject matter expertise (e.g. epidemiological and research methods), facilitation of training, facilitating linkages of placement site staff with appropriate resources, and the sharing of lessons learned from previous placements.

Public Health Officers have also facilitated networking for site organizations in order to strengthen specific projects or to better integrate them with other public health organizations, both regionally and nationally. For example, Public Health Officers indicated that they have broadened access to networks of expertise for the site through their own existing contacts, and have also established new relationships that may be of use for the site after the placement has finished. The growth of existing networks and the establishment of new ones have served, in some cases, as a tool to promote inter-jurisdictional connections and multi-disciplinary approaches to address public health challenges.

Sustainability of benefits

A major issue that was raised by many site supervisors and Public Health Officers is the sustainability of the capacity that each Public Health Officer brings to a site once the placement has ended. Interviewee comments indicate a widespread feeling that this capacity might potentially be lost once the Public Health Officer has been rotated to another site, unless deliberate efforts are made during their tenure to build capacity in more lasting ways. In a number of cases it was clear that Public Health Officers have indeed made deliberate efforts to sustain capacity through knowledge transfer to core staff through training and tool development, the production of public health evidence that can be used for decision-making (e.g. epidemiological overview of sexually transmitted infections in Nunavut), and the creation of enduring partnerships. Some interviewees and survey respondents also mentioned that the presence of the Public Health Officer can demonstrate the value of an additional staff position to placement site authorities which may eventually encourage the creation of a new locally-funded public health position.

This section reviews the extent to which the Public Health Agency has achieved its intermediate outcome related to the development of a cadre of Public Health Officers, available to function in a range of public health issues, including mobilizations.

Enhanced ability of Public Health Officers to function in a range of public health issues, including mobilizations

Public Health Officers have worked on projects covering a spectrum of public health topics (Table 4 in section 3.2.4), have received group training to prepare for potential mobilizations, and in some cases, have had the opportunity to mobilize. Currently the program has a cadre of 14 Public Health Officers that have been engaged in a range of issues and have also been mobilized on short term assignments. 

Despite a lack of complete documentation for the 42 placements of Public Health Officer to date, information available shows that Public Health Officers have worked on projects covering a spectrum of public health topics, including emergency management, environmental health issues, infectious disease issues, public health policy support, reproductive / sexual health and the development of surveillance infrastructure. At least five cases were noted where Public Health Officers were engaged on more than one topic within a given placement.

In addition to exposure to a range of public health topics, there have also been nine instances in which Public Health Officers were mobilized to respond to emerging public health issues outside of the site placements (Table 5). Between 2006 and 2013, there were ten mobilizations of Public Health Officers to work on issues related to surveillance of outbreaks and public health infrastructure. Most mobilizations were less than one month in length. While most mobilizations initially occurred within Canada, international mobilizations have occurred more recently. The international nature of recent mobilizations was based on need and Public Health Officer availability rather than a program shift towards international mobilization.

The breadth of public health topics covered in placements and the number of mobilizations that took place indicate that Public Health Officers represent a growing resource of public health expertise within Canada. However, difficulties recruiting and retaining Public Health Officers have hampered the realization of the vision of a cadre of qualified and capable Public Health Officers that is available for deployment. Despite these recruitment and retention challenges however, which are discussed in greater detail later in the report, the majority of Public Health Officers that have left the program remain employed in the field of public health. In these roles, these former Public Health Officers continue to use the skills and knowledge they gained from the Canadian Public Health Service program, and they continue to contribute to Canada’s public health capacity needs.

This section reviews the extent to which the Public Health Agency has achieved its intermediate outcomes related to the development of Public Health Officers.

Enhancement of Public Health Officer knowledge and skills

Evaluation evidence indicates that the Canadian Public Health Service program has made progress towards its intended intermediate outcome of developing transferable core competencies in Public Health Officers. While the program has not yet established official core competency training or related performance measurement materials, current and former Public Health Officers indicated that they have acquired transferable knowledge and skills as a result of their site placements. Public Health Officers indicated that each individual project provides them with the opportunity to gain knowledge of a specific public health topic; however, Public Health Officers reported that they have also gained:

• knowledge of the federal government’s role in public health (including an understanding of federal-provincial-territorial relations) and the practice of public health in Canada and the stakeholders involved
• knowledge of technical tools and processes (e.g. emergency preparedness and response, surveillance and analysis in specific subject matter areas)
• skills related to working in diverse public health organizations (e.g. collaboration with partners and stakeholders).

In general, the views of Public Health Officers were corroborated by feedback from Regional Coordinators, site supervisors, and program management. The acquisition of knowledge and skills was facilitated through mentoring, training, and hands-on experience they received during their placement.

Public Health Officers received mentoring from Regional Coordinators, site supervisors, and program managers. The mentoring efforts of Regional Coordinators were highlighted by the Public Health Officers as being particularly beneficial. Public Health Officers spoke highly of the collegial relationships formed with their Regional Coordinator and the importance of having them as a point of contact to discuss both their program experiences and requirements. The mentoring provided by site supervisors was also identified by Public Health Officers as being useful, particularly with respect to providing linkages with local networks and technical advice/feedback. This was especially important in rural and remote sites. Despite the physical distance between Public Health Officers and program management, the mentoring provided by program managers was also identified by Public Health Officers as beneficial, particularly with respect to technical program requirements. However, it should be noted that in the Public Health Officer survey, the mentoring provided by the Regional Coordinators was consistently rated as the most beneficial source of mentoring.

Public Health Officers also referenced the training they received as contributing to the enhancement of their knowledge and skills. A number of group training opportunities were provided by the Public Health Agency, including the Field Services Training Institute and training on public health core competencies, which recently focussed on emergency response. Some Public Health Officers with epidemiological backgrounds were also provided the opportunity to participate in the ‘Epi in Action’ training, which prepared them for potential mobilizations. In addition to group training, individuals have been able to access training specific to their needs, such as advanced training on technical skills (e.g. statistical software and geographic information systems) and participation in relevant professional conferences. The role of Regional Coordinators, site supervisors, and program management in facilitating access to training opportunities was highlighted by Public Health Officers. 

Lastly, hands-on experience during their site placements was mentioned as a means by which Public Health Officers have benefited from the program. In general, Public Health Officers identified the practical experience in applying public health knowledge and skills as key, especially for those early in their public health career. Public Health Officers were also challenged to exercise general aptitudes (e.g. identifying needs, taking initiative and collaborating with others), and benefited from the variety of work experiences across different organizations and regions. Although the majority of Public Health Officer survey respondents indicated they worked on emerging public health issues during their placements, many respondents had anticipated more opportunities to be engaged. This presents an opportunity for further engagement of Public Health Officers as a surge capacity resource.

Establishment of Public Health Officer professional networks

While not listed as an intended immediate or intermediate outcome, feedback from Public Health Officers indicates that site placement experiences and the collegial nature of the program have facilitated the development and use of professional networks. This includes strengthened links with sources of public health expertise in academia and at the Public Health Agency and supportive relationships with fellow Public Health Officers. General observations from Public Health Officers imply that the development of personal networks form part of an increased understanding of the practice of public health nationally and in different regions. Moreover, such networks can be used to inform work in subsequent placements or public health positions outside of the program.

This section reviews the extent to which the Public Health Agency has achieved its intermediate outcomes for Canadian Public Health Service Students.

Experienced gained by students

Between 2006 and April 2013, 30 students undertook a practicum through the Canadian Public Health Service program, of which 29 had successful completion. Students were placed in the Northwest Territories (n = 15), Nunavut (n = 6), Yukon (n = 5), and Labrador (n = 2). Table 6 shows the detailed distribution of the placements.

Evidence gathered during the evaluation indicates that the work experience gained by students through participation in the program has fulfilled educational requirements for graduate degrees in public health, and has stimulated an interest in pursuing public health careers in the North. It has also been noted by interviewees that, following their placements, a few students have been hired into other positions, either federally, provincially or locally, although precise numbers and locations are not available.

In general, the documentation from student placements, as well as interviews and case studies, suggests that the majority of students feel their placement allowed them to:

• enhance their knowledge and skills in a practical setting that complemented their academic studies and served as an opportunity to confirm their choice of public health as a career path
• be exposed to the public health context, work and living conditions of northern communities
• be introduced to stakeholders and networks, such as community representatives and public health experts, as well as other students
• fulfill their educational requirements for graduation.

This section describes reported challenges identified during the evaluation.

Duration of Public Health Officer placements

Multiple stakeholder groups (Public Health Officers, site supervisors, senior provincial and territorial representatives) expressed concern that the Canadian Public Health Service program is not flexible enough in its design, particularly related to the duration of Public Health Officer placements. They feel it has limited the responsiveness of the program. The majority of key informants feel that site placements should be longer in duration (three years was most commonly recommended) to increase the likelihood that project work will be completed, and thereby more effectively address site needs.

Site supervisors explained that although the Public Health Officer is placed in their site for two years, the actual amount of time spent by the Public Health Officer on the project is significantly less. Upon arrival at their placement site, Public Health Officers must first be oriented and trained, which, according to site supervisors, takes approximately six months. Further reducing the actual amount of project time is that towards the end of the placement, Public Health Officers become involved in the process for their next rotation. This involves time away from their current placement for interviews with potential host organizations and house hunting trips. While the majority of stakeholders supported longer placements, the consensus was that it would be most appropriate to link the placement duration with the Public Health Officer project work. Interviewees argued that this would help improve the program’s responsiveness as Public Health Officers could be rotated as soon as their project work was complete (even if they have been in their site for less than two years).

The current design of the Canadian Public Health Service program also appears to have negatively impacted program outcomes for both Public Health Officers and placement sites. Current and former Public Health Officers reported that in the past, they have had to rotate from their placement site prior to completing their project work. They feel that this has truncated their learning in some instances, as they have not been able to see projects through to completion in all cases. Similarly, the departure of Public Health Officers at the two year mark has left some placement sites with incomplete project work, which in some cases the site has not been able to complete in the absence of the Public Health Officer. It is important to note however that with improved communication materials from the program in recent years, wherein the fixed duration of the placement is clearly outlined, this has been somewhat less of a challenge as site applications include projects of shorter durations.

Rotation process

Evaluation evidence collected from review of program documentation, as well as feedback from key informants, describes how the policy around Public Health Officer rotation has changed over the course of the Canadian Public Health Service program.^{Endnote 7}, ^{Endnote 9} While the intention to rotate Public Health Officers had been envisioned since the program’s inception, the operationalization of this intention has evolved over time. Public Health Officers and site supervisors originally understood that rotation was only a potential occurrence, however, program policy and documentation has now been revised to describe the mandatory nature of rotation. Additionally, rotation policies have become increasingly more prescriptive in that they now require Public Health Officers to rotate to a different province.

Through both surveys and key informant interviews, current and former Public Health Officers explained that the requirement to move not only themselves, but often their families as well, to a completely different province every two years is a less than ideal situation. They have found this all the more stressful given that they may not receive one of their top three choices during the matching process. This situation can potentially result in the Public Health Officer and their family having to relocate to a community that may not meet their unique needs (e.g. medical, daycare, etc). Current and former Public Health Officers expressed that this approach has contributed to a sense of impermanence in their professional and community lives.

While the rotation process has several associated challenges, it is not without its benefits as well. Current and former Public Health Officers indicated that they have benefitted from rotation through exposure to how public health is practiced in multiple jurisdictions. Opportunities to work in various sites have also contributed to increased Public Health Officer learning and have facilitated knowledge exchange which has, in turn, benefitted placement sites.

It is important to note that several aspects of the current design and delivery model appear to have contributed to a smaller than envisioned cadre of Public Health Officers. The program had originally planned for an on-going cadre of approximately 40 Public Health Officers. At its staffing peak the program employed 27 Public Health Officers. Various staffing restrictions have resulted in temporary hiring freezes and an inability to backfill vacant Public Health Officer positions. This in turn has resulted in a steadily declining number of Public Health Officers which is limiting the ability of the program to most effectively carry out its mandate. Currently there are only 14 active Public Health Officers. Retention of PHOs has also been an on-going issue. Although there have been occasions where Public Health Officers have left the program to pursue academic or employment opportunities, this has not frequently been the case. According to exit interviews conducted with Public Health Officers at the time of their departure from the program, the most common reason given (approximately 80% of Public Health Officers) for their exiting the program is due to their displeasure with the requirements associated with rotation.

Another factor, perceived by program stakeholders to be contributing to decreased Public Health Officer retention, is the lack of opportunity for career advancement within the program. Public Health Officers are hired as indeterminate employees and, for as long as they remain in the program, they remain at their current classification level. In other words, if a Public Health Officer desires to move forward in their federal public service career, they must leave the Canadian Public Health Service program. Returning to the Public Health Agency however can be problematic as Public Health Officers placed outside of the National Capital Region (where the majority of Public Health Agency positions are located) may not be eligible to apply for many of the posted competitions in the National Capital Region. Of the 10 Public Health Officers that have left the program, only 1 is still employed by the Public Health Agency. These issues have negatively contributed to the development of a Public Health Officer cadre, as discussed in section 3.2.5.

3.3Efficiency & Economy

The Treasury Board Policy on Evaluation (2009) defines the demonstration of economy and efficiency as an assessment of resource utilization in relation to the production of outputs and progress toward expected outcomes.  This assessment of economy and efficiency is based on the assumption that departments have standardized performance measurement systems and that financial systems use object costing.

Given the lack of departmental financial data that is linked to the quantity and type of outputs coupled with incomplete program-level expenditure reporting, the evaluation could not conduct an assessment of resource utilization in relation to the production of outputs and progress toward expected outcomes.

As a result, this evaluation provides broad observations on economy and efficiency based on the document review, literature review and key informant perspectives. More specifically, observations are provided on overall expenditures and program delivery efficiency. In addition, observations are made regarding the adequacy of performance measurement data, and opportunities for improved collaboration.

Observations on efficiency and economy

Variation in the expenditures during the early years of the program reflects the development nature of the program at that time. Over the last three fiscal years, the Canadian Public Health Service program was budgeted to spend $3.0 to $3.5 million annually. Of this amount, approximately $800 thousand annually was budgeted for transfer to Regional Operations to support their involvement in the program. In any given year, salary expenditures represent the largest portion of program costs and are linked to costs for Public Health Officers, graduate students, program management, and Regional Coordinators. This is followed by expenditures for operations and maintenance. Costs related to relocation (i.e. placement of new Public Health Officers, mobilization of existing Public Health Officers, and placement of graduate students) account for the remainder of program expenditures (Table 7). Although initially there was a grant and contribution component to fund graduate student practicums, it was cancelled in 2010.

Source:Public Health Agency, Office of the Chief Financial Officer

In general, variations from planned expenditures vary over time. The largest variances occurred in the early years of the program (2007-08 and 2008-09) while initial staffing was underway. During these years the program underspent by $1.2 to 1.7 million. In contrast, during a year of full staffing and relocations (2010-11) the program overspent by approximately $120,000.

Variations in salary expenditure are related to the fluctuations in the number of Public Health Officers in the field. As seen in Table 8, salary expenditures were highest in 2010-11 and 2011-12, when there were more Public Health Officers in the field. Relocation expenditures are also higher during years with more activity.

Reimbursement for relocation expenses result in similar costs to other government departments who relocate staff. In 2012-13 fiscal year, the cost of individual rotations ranged from $17,000 to $45,000. This is similar to the Royal Canadian Mounted Police, who estimated the average cost over 2,100 relocations in 2008 –09 to be $30 – 35,000.^{Endnote 34} The wide range reflects the distance between sites and the size of the household being relocated. Of note, the average cost per relocation was not calculated because data from Public Health Officer relocations, graduated student practicums, and mobilizations were combined in the financial tracking system. These events include different costs and would not have provided valid estimates.

Despite an inability to report on the specific program related costs in comparison to the achievement of program outcomes, the evaluation noted some program efficiencies. One example of this was the use of group training to address commonalities in public health skills required by Public Health Officers including emergency preparedness.

There is minimal performance measurement in place to help the Office of Public Health Practice monitor progress and assist in programmatic decision making. The program does not have a management approved logic model summarizing the program theory, nor an associated narrative, as discussed in section 3.2.2.

Performance measurement data

The Canadian Public Health Service program does not have a formal performance measurement strategy in place to monitor progress and assist programmatic decision making. However, there was minimal performance data available for analyses for this evaluation.

While limited performance data was available about program outcomes, the evaluation team has been able to conduct an analysis of the performance measurement data collected about program activities and outputs. The program tracked some information relating to the selection of placement sites and the activities of various program participants (including Public Health Officers, graduate students, and site supervisors). For example, with the support of Regional Coordinators, some site placement data was collected at the six and eighteen month marks as well as at the end of each placement. This information included feedback from the Public Health Officers on the placement process and workload, as well as on the availability of onsite training and supervision. In addition, the program collected information on the student placement via assessment forms designed by the university to which the student attended. 

Although some performance data was collected, the evaluation team determined that this performance information was not tracked consistently, and gaps in the data available presented challenges for the analyses of program activities, outputs and outcomes. 

There is also no evidence that the performance information collected was ever used to inform program or policy decisions regarding the Canadian Public Health Service program. Performance information could have been better used to assess and adjust performance for the program as a whole.

According to various internal interviews and documents, collaboration with internal stakeholders is required to support the Canadian Public Health Service’s activities to develop well established networks and partnerships with key stakeholders. Finding an appropriate level of engagement with internal stakeholders has been an ongoing challenge. Opportunities for enhanced collaboration were identified.

Linkages within the Public Health Agency

Since the program’s inception in 2006, the Canadian Public Health Service program staff has initiated dialogue to share information and collaborate with a number of other areas within the Office of Public Health Practice and other Public Health Agency colleagues on related work.

There has been collaboration and partnering to deliver a variety of training opportunities to Public Health Officers and other Public Health Agency staff working on related issues. Within the Field Services Training and Response section, there is a Field Services Training Unit that supports professional development for both the Canadian Public Health Service program and the Canadian Field Epidemiology program staff. Over the past few years, a number of joint training offerings have been available which serve both programs as well as other Public Health Agency staff working in related fields. Examples of training programs that have been offered include: the Field Services Training Institute and the Epidemiology in Action courses.

The working relationship between Canadian Public Health Service program staff in the National Capital Region office and the various Public Health Agency regional offices has been key to the effectiveness of the delivery of the program, where ongoing liaison between these offices has been a critical element in the design and delivery of this program. Regional Coordinators have been responsible for liaison with Public Health Officers and graduate students at their local placement sites. In addition, Regional coordinators have helped promote the program locally and supported the initial selection of sites and monitoring of activities at placement sites. Furthermore, through feedback from interviews and surveys, current and former Public Health Officers and graduate students underlined the importance and value of their link with Regional Coordinators.

Opportunities for engagement within the Public Health Agency

A variety of interviewees, as well as the case studies, suggested that there is limited collaboration and knowledge exchange between the Canadian Public Health Service program participants and Public Health Agency staff working in similar content areas. There have been commonalities between the subject matter of projects undertaken by Public Health Officers at their placement site (e.g. sexual health strategy in the North) and work being undertaken by other groups within the Public Health Agency (e.g. Centre for Communicable Diseases and Infection Control).

This finding is supported by feedback from Public Health Officers who indicated that they did not have a good understanding of opportunities to link with specific Public Health Agency issue experts to support them in their placement activities.  Public Health Officers indicated that they felt a closer connection with their host organization rather than with their “federal family” at the Public Health Agency. At their placement site, while the Public Health Officers often linked with Public Health Agency staff assigned to placements through the Canadian Field Epidemiology Program, they indicated that they were unaware of other field services staff working in their province or region.

To help improve program efficiencies, interviewees suggested additional opportunities for Public Health Officers to engage with these other program areas within the Public Health Agency. They argued that this would enhance links between public health priorities and the Canadian Public Health Service program. This effort could leverage shared knowledge and partnerships among program areas, leading to more efficient and effective delivery of program activities. Public Health Agency program areas suggested for additional engagement include those inside the Office of Public Health Practice, such as the Canadian Field Epidemiology Program, as well as those within other branches including the Infectious Disease Prevention and Control Branch, Health Promotion and Chronic Disease Prevention Branch, and the Emergency Management and Regulatory Affairs Branch. 

In summary, there are some examples of collaboration with some key internal partners, particularly with respect to training and engagement with regional colleagues. However, collaboration appears to be largely absent with other potential internal partners. The impact of this gap on the Public Health Agency’s ability to achieve the objectives of the Canadian Public Health Service program in the most efficient manner possible is not known. It is anticipated that greater integration and collaboration would support greater leveraging of knowledge and enhance the effectiveness of the program.