Section 4: Evaluation of the Canadian Public Health Service Program at the Public Health Agency of Canada – Conclusions and recommendations

4. Conclusions and recommendations

The synthesis and analysis of the findings from this evaluation resulted in conclusions about the relevance and performance of the Canadian Public Health Service program. In turn, this led to three recommendations for senior management of the Public Health Agency.

4.1Conclusions

4.1.1Relevance

Although recent empirical data is limited, there are commonly accepted public health capacity gaps in Canada. In particular, there are regional disparities in public health capacity across the country. Regions with acknowledged capacity needs include northern (territories) and eastern Canada (Atlantic provinces) as well as rural/remote areas of all provinces. Specific capacity needs in these regions often relates to the recruitment and retention of trained public health professionals. The most common needs by profession include epidemiologists, nurses, and policy analysts. Together, these gaps represent critical shortfalls in Canada’s public health system and could contribute to a reduced capacity to respond to a public health event. There is therefore a continued need to address Canada’s public health capacity gaps.

Building Canada’s capacity in public health is an appropriate role for both the federal government and the Public Health Agency.  While provinces and territories have the primary responsibility for their respective human resource and capacity needs, the consequences of their capacity gaps can affect the public health system as a whole. As such, the Government of Canada may choose to play a supporting role in this situation. The Public Health Agency’s role in addressing Canada’s public health capacity needs generally aligns with and supports the Agency’s broader public health mission “to promote and protect the health of all Canadians, through leadership, partnership, innovation and action on public health.”  Similarly, the Public Health Agency’s role includes: intergovernmental collaboration on public health; the facilitation of national approaches to public health; emergency preparedness and response; and the prevention and control of chronic and infectious diseases. Through the Canadian Public Health Service activities, the Agency is performing this role. The Canadian Public Health Service program does not appear to duplicate the role of other stakeholders. However, the Canadian Public Health Service program has the potential to overlap with the Canadian Field Epidemiology Program, in particular if the focus on training and development of Public Health Officers is prioritized over building site capacity.

Strengthening Canada’s public health capacity is a priority for the Government of Canada and the Public Health Agency and is encompassed within three of the latter’s strategic directions for the next five years, as per corporate strategic documents: leadership on health promotion, disease prevention, and health protection; strengthened public health capacity and science leadership; and enhanced emergency preparedness and response.

4.1.2Performance

The Canadian Public Health Service program has made progress towards its stated intermediate outcome of helping to address public health gaps at participating placement sites.  Through Public Health Officer placements, sites gained an improved ability to address ongoing core public health work, an improved ability to undertake self-identified priority projects that they were previously unable to pursue, and an improved response to public health events.  Similarly, student projects supported organizational capacity gaps by allowing sites to undertake discrete projects that may not otherwise have been accomplished in the desired timeframe.

Evaluative evidence also indicates that the Canadian Public Health Service program has made progress towards achieving its stated intermediate outcome of enhancing capacity of partner organizations to fulfill their public health needs. Site supervisors reported that Public Health Officers have contributed to their sites through the provision of technical skills such as epidemiological surveillance and analysis, the development of high-quality reports, the facilitation of networking, as well as the delivery of training to site staff. Public Health Officers have also facilitated knowledge transfer through their sharing of lessons learned from previous site placements.

Although the sites have benefitted from the placement of a Public Health Officer in their organization, it appears that this capacity might potentially be lost, in some instances, once the Public Health Officer has been rotated to another site unless deliberate efforts are made during their tenure to build capacity in more lasting ways.

Progress has also been made towards achieving the intermediate outcome related to the development of a cadre of Public Health Officers, available to function in a range of public health issues, including mobilizations. Public Health Officers have worked on projects covering a spectrum of public health topics, have received group training to prepare for potential mobilizations, and in some cases, have had the opportunity to mobilize.

Similarly, the program has also made progress towards achieving its intermediate outcomes related to the development of Public Health Officers. While the program has not yet established official core competency training or related performance measurement materials, current and former Public Health Officers indicated that they have acquired transferable knowledge and skills as a result of their site placements. Furthermore, students have fulfilled educational requirements for graduate degrees in public health, and have gained work experience in public health in the North.

While the program has made progress towards the achievement of the intermediate outcomes currently stated in its draft logic model, the evaluation identified that significant challenges remain with respect to the program’s overarching design and delivery. The program’s intended results and associated program theory are unclear to many program stakeholders. As a result, the site selection and matching process appear to have resulted in placement sites being selected that do not represent those within the country with the greatest public health capacity needs, as indicated by senior provincial/territorial representatives during data collection. Additionally, the lack of flexibility in the duration of Public Health Officer placements and the requirement to rotate has negatively impacted program outcomes for placement sites and Public Health Officers. The requirement to rotate has also led to decreased Public Health Officer retention.

Finally, additional challenges remain, including the missed opportunity for substantial collaboration and knowledge exchange between the Canadian Public Health Service program staff and Public Health Agency staff working in similar content areas.  This collaboration could lead to leveraging of shared knowledge and partnerships among program areas, resulting in a more efficient and effective delivery of program activities.  Lastly, there lack of a performance measurement strategy to help the Office of Public Health Practice monitor progress and assist in programmatic decision making, and therefore performance information could have been better used to assess and adjust performance for the program as a whole.

4.2Recommendations

4.2.1Intended Outcomes

The Public Health Agency aims to enhance public health capacity in Canada through a number of field service programs. Each of these programs targets a specific component or element of the public health system and employs its own unique approach to building capacity. While work is currently underway within the Agency to define and describe what is meant by public health workforce and capacity, at the time of data collection, there did not appear to be a commonly accepted Agency-level understanding of this terminology. Therefore, the precise means by which the Canadian Public Health Service program compliments the Agency’s other efforts in this area is not clearly articulated.

Evaluation evidence indicates that the Canadian Public Health Service program objectives are not clear or well understood by all program stakeholders. This may be the result of a number of factors. Firstly, the authorities for this program covered Avian and Pandemic activities more broadly, and therefore limited details were included with respect to the specific intent and rationale for the Canadian Public Health Service. Secondly, there is a limited availability of detailed program documents (strategic and descriptive). Thirdly, the program does not have a management approved logic model summarizing the program theory. A logic model was provided for the purpose of the evaluation; however, it is incomplete as the two ultimate outcomes for this program were still under development at the time this evaluation was being conducted. Further, the logic model identifies general program outcomes (immediate, intermediate and ultimate) but does not clearly identify program inputs, activities or outputs. Additionally, the draft logic model does not contain clearly defined outcomes. For example, the ultimate outcome of “strengthening public health capacity” is interpreted differently by various program stakeholders.

This lack of clarity in the program theory has negatively impacted the delivery of the program and thus its ability to achieve the most appropriate results for the Agency within the context of the other field service programs.

Recommendation 1

Within the context of the Agency’s strategic directions and efforts towards building public health capacity, determine, articulate and communicate desired program results.

4.2.2Design and Delivery

The current lack of clarity associated with the program’s intended outcomes has resulted in a program delivery model that lacks a clear, consistent and commonly understood strategic approach.  This is best demonstrated by the site selection criteria that are used to assess placement site applications. These site selection criteria are not well defined and have undergone multiple iterations since program inception. In addition, they appear to be paradoxical, as the current program delivery model may not support the achievement of both intermediate outcomes. The site selection criteria also do not appear to reflect or support all the activities outlined in the original program mandate (e.g. surge capacity), and may also be missing links to current program priorities.

The site selection criteria have also partly contributed to sites being selected that do not appear to represent those within the country with the greatest perceived public health capacity needs. While Regional Coordinators have liaised with provincial/territorial and local stakeholders to identify their capacity needs/priorities via the solicitation of proposals, feedback from senior provincial/territorial representatives suggests that the program is still not reaching the provincial/territorial areas with the greatest capacity needs. The current placement of Public Health Officers has reflected the profession-related public health capacity needs (epidemiology, public health nurse, policy analyst); however, they do not reflect recognized regional disparities (Northern communities, Atlantic provinces, rural/remote areas).  Furthermore, the majority of Public Health Officer placements to date have taken place in larger urban centres, as opposed to the priority rural and remote areas identified by key informants.

Similarly, the current duration of Public Health Officer placements and the requirement to rotate may not support the achievement of desired program outcomes. For example, the requirement to rotate Public Health Officers at the two year mark has resulted in incomplete project work at some placement sites, leaving these sites with ongoing capacity needs. Likewise, the requirement to rotate at the two year mark has also negatively impacted learning and development outcomes for Public Health Officers and slowed the formation of a sustainable cadre of Public Health Officers due to decreased Public Health Officer retention.

By articulating a clear strategic focus and developing accompanying operational processes, the program will be positioned to align its resources to more effectively achieve priority outcomes. This may partly be a result of the requirement for placement sites to have strong mentoring capacity to also facilitate Public Health Officer professional development.

Recommendation 2

Review and revise current operational practices, including approaches to site selection, rotation, and placement duration, to support achievement of desired results.

4.2.3Efficiency and Economy

Performance Measurement

The Canadian Public Health Service program does not have a formal performance measurement strategy in place to monitor progress and assist in programmatic decision making.  This is exacerbated by the lack of a detailed, management approved, logic model and associated narrative clearly describing program theory.

Limited performance data has been collected about program outcomes, and the data collected about activities and outputs appears to be inconsistent and unsystematic. Gaps in the performance data present challenges when attempting to measure and assess the program’s progress towards the achievement of its intended outcomes for Public Health Officers, students and placements sites. There is also no evidence that the limited performance information collected was ever used to inform program or policy decisions regarding the Canadian Public Health Service program.

There are promising activities currently in the works which hint at the development of a more comprehensive performance measurement culture within the Canadian Public Health Service program.  For example, the program is working to develop a more substantive logic model and narrative which would clearly articulate the proposed theory of change. While this is a positive indication of a more effective performance measurement function, future work must be comprehensive in its approach (documenting outputs and outcomes of both Public Health Officer and student placements), grounded in a clear theory of change, systematically integrated into operations, and then actively used in management practices to assess and adjust performance.

Collaboration

Since the program’s inception in 2006, the Canadian Public Health Service program staff has initiated dialogue to share information and collaborate on related work with a number of other areas within the Office of Public Health Practice and other Public Health Agency colleagues.

There has been significant collaboration with the regional officers, including partnering to deliver a variety of training opportunities to Public Health Officers and other Public Health Agency staff working on related issues. The working relationship between Canadian Public Health Service staff in the National Capital Region office and the various Public Health Agency regional offices has been key to the effectiveness of the delivery of the program, and ongoing liaison between these offices has been a critical element in the design and delivery of this program.

Collaboration with other areas of the Agency has been minimal.  For example, there is limited collaboration and knowledge exchange between Public Health Officers / students and Public Health Agency staff working in similar content areas, and Public Health Officers do not have a good understanding of opportunities to link with specific Public Health Agency issue experts to support them in their placement activities.

The impact of this gap on the Public Health Agency’s ability to achieve the objectives of the Canadian Public Health Service program in the most efficient manner possible is not known. It is anticipated that greater integration and collaboration would support greater leveraging of knowledge, and enhance the effectiveness of the program.

Recommendation 3

Maximize opportunities for enhanced program performance by:

• a)Strengthening the performance measurement approach for the program to better inform programmatic
         decision-making.
• b)Facilitating collaboration and knowledge exchange activities across the Public Health Agency of Canada.