Evaluation of Community Associated Infections Prevention and Control Activities at the Public Health Agency of Canada

Prepared by
Evaluation Directorate
Public Health Agency of Canada/Health Canada

February 2013

Table of contents

• Executive summary
• 1. Introduction
  • 1.1 Purpose and scope of the evaluation
  • 1.2 Methodology
• 2. Background and context
• 3. Findings
  • 3.1 Relevance
  • 3.2 Performance
• 4. Conclusions and recommendations
• Appendix A. Logic model and organization chart
• Appendix B. Knowledge products
• Appendix C. Visitor Profile
• Appendix D. Comparison: surveillance reporting
• Appendix E. History of collaboration with the Public Health Network
• Appendix F. Financial expenditures for the Hepatitis C Program
• Appendix G: ACAP, Hepatitis C Program Outcomes
• Bibliography

Table of figures

• Figure 1: Findings, conclusions and recommendations
• Figure 2: Key informant interviews
• Figure 3: Limitations
• Figure 4: Activities and outputs of the Surveillance and Epidemiology Division
• Figure 5: Activities and outputs of the Professional Guidelines and Public Health Practice Division
• Figure 6: Activities and outputs of the Programs and Partnerships Division
• Figure 7: Activities and outputs of the Strategic Issues and Integrated Management Division
• Figure 8: Outputs leading to immediate and intermediate outcomes
• Figure 9: National surveillance data for chlamydia, gonorrhea, syphilis, hepatits B and C, and tuberculosis
• Figure 10: The top 10 communicable disease web pages for the period of April 1, 2010 to March 31, 2011
• Figure 11: Populations targeted by hepatits C projects
• Figure 12: Number of individuals reached versus those reporting increased knowledge of hepatitis C transmission and risk
• Figure 13: Evidence of “use of knowledge” of selected Public Health Agency documents
• Figure 14: Presents how survey respondends and key informants describe the use of Public Health Agency’s products in their work
• Figure 15: Improved access to health services
• Figure 16: Policy change
• Figure 17: Intention to change behaviour
• Figure 18: Summary of the Centre for Communicable Diseases and Infection Control’s key partners and partnership mechanisms
• Figure 19: Hepatitis C project partnerships by sector
• Figure 20: Hepatitis C project partnerships by discipline

Executive summary

This report presents the findings of an evaluation of prevention and control of community associated infections activities at the Public Health Agency of Canada.

Evaluation purpose

This evaluation was conducted to support program planning and decision making. Evaluative information was also needed to assess funding received to address hepatitis C in Canada. The evaluation covers activities undertaken between 2008 and 2012.

The evaluation does not take into consideration activities related to the National Microbiology Laboratory, the Canadian HIV Vaccine Initiative, the Federal Initiative to Address HIV/AIDS, the Quarantine Program and the Travel Health Program. It also excludes healthcare associated infections activities that may be covered by the Centre for Communicable Diseases and Infection Control at the Public Health Agency. These activities will be assessed in upcoming evaluations.

Description of activities

The Public Health Agency defines ‘community associated infections’ as communicable diseases transmitted person-to-person in the general community context (as opposed to health care settings). This would include hepatitis B and C, sexually transmitted infections such as syphilis, gonorrhea and chlamydia, HIV/AIDS, and tuberculosis. As mentioned above, the Public Health Agency’s activities in the Federal Initiative to Address HIV/AIDS will be covered in a separate evaluation.

There are many players addressing community associated infections in Canada. Provinces and territories, other government departments, local community groups as well as academics and researchers all have key roles in preventing and controlling these types of infections.

Within the Public Health Agency, work to prevent and control ‘community associated infections’ is led by the Centre for Communicable Diseases and Infection Control. The mandate of the Centre for Communicable Diseases and Infection Control is to create and share credible knowledge and facilitate coherent national action which contributes to the prevention and control of specific communicable disease, with a focus on key populations at risk.

This evaluation examines the community associated infections role and activities of the Centre for Communicable Diseases and Infection Control at the Public Health Agency. The Agency allocated about $12.1 million for these activities in 2012-13.

Evaluation findings, conclusions and recommendations

The synthesis and analysis of the findings from this evaluation resulted in conclusions about both relevance and performance. In turn, this led to three recommendations for Public Health Agency senior management.

Findings and conclusions

Relevance

The need to prevent and control community associated infections (namely hepatitis B and C, sexually transmitted infections, and tuberculosis) still exists. Economies can be achieve and quality of life can be improved by targeting prevention and control activities towards common populations, risk factors and social conditions underlying each disease. Disease-specific work is beneficial in many circumstances, for example, when distinct target groups exist and in areas such as public health practice, surveillance (without precluding the importance of integrated surveillance systems addressing issues of coinfection), vaccine development and biomedical research.

Infectious disease prevention and control is an integral part of the Public Health Agency’s role and mandate. With the exception of a few areas such as infection control at the Canadian border, the majority of this mandate is complementary and shared with provincial/territorial governments and other public health stakeholders in Canada. As such, the Public Health Agency’s role must provide added value within a system of multiple players. Key to this role is providing a national picture of infectious disease in Canada, aligning and coordinating national efforts along provincial and territorial directions and priorities, and filling important gaps in research, knowledge and program or policy directions that would not otherwise be addressed.

Infection control is a priority for the Public Health Agency, and is encompassed within all three of the Public Health Agency’s strategic directions for the next five years, as per corporate strategic documents: leadership on health promotion, disease prevention, and health protection; strengthened public health capacity and science leadership; and enhanced emergency preparedness and response. Within these strategic directions, tuberculosis and sexually transmitted infections (along with antimicrobial resistance) were highlighted as priorities by most internal key informants and provincial and territorial partners.

Performance

Achieving an appropriate level of collaboration with provinces, territories and stakeholders is a key outcome of the community associated infections activities in the Centre for Communicable Diseases and Infection Control. Increased awareness on the state, incidence, trends of communicable diseases and specific public health communicable disease practices in Canada as well as the increased use and application of the evidence, knowledge and best practices are also key outcomes.

The evaluation found that the Public Health Agency works collaboratively with provinces and territories as well as some stakeholders in their activities to prevent and control community associated infections.  Mechanisms are in place to share information, to collaborate through working groups, reviewing guides and providing input. Experts in the area are engaged to develop guides.

The Agency has also encouraged partnerships through funded community projects seeking to address hepatitis C locally. Partnerships were fruitful in helping funding recipients achieve their project objectives more efficiently (i.e. using fewer resources) and more effectively (i.e. having greater impact).

There are also indications that Public Health Agency products increase knowledge in the area of community associated infections and are being used by partners and stakeholders. Assessments of some tuberculosis, sexually transmitted infections, sexual health and hepatitis C products demonstrate that there is increased knowledge after usage. However, there are also signs that increased knowledge could be at a threshold for some community-based projects.

There is documented evidence of the use of Agency products. For example, when the Tuberculosis online modules were evaluated, doctors responded to questions on how they will change their practice. Several groups reported that they would be more vigilant in screening and are more aware of risk groups. Medical schools have used the Sexually Transmitted Infections guidelines/online modules for instructional purposes. Key informants also noted that Public Health Agency products have been used for instruction, for development of guidelines in other countries and to improve public health practice.

However, there are opportunities for improvement. The Public Health Agency could have extended its collaboration network with specific stakeholders and provided national information in a timely fashion, which may have impacted the usefulness of some Agency products. In addition, there is limited performance measurement in place to help the Centre for Communicable Diseases and Infection Control monitor progress and assist in programmatic decision making.

Recommendations

Collaboration

The Public Health Agency shares jurisdiction over public health and health promotion with numerous stakeholders. Each player has a different role and makes a unique contribution to the overall Canadian effort. Through collaboration, the Public Health Agency can reduce duplication of effort and maximize complementarity of programming.

To prevent and control community associated infections, the Centre for Communicable Diseases and Infection Control’s primary partners are: other federal government departments, provincial and territorial governments, non-governmental organizations, researchers and academics, and international agencies. Collaboration tends to be working well with provinces and territories as well as some key internal partners, which is important during an outbreak. However, collaboration with other partners could be better sustained. While there were indications of some positive activities with other government departments, some survey respondents, including provinces and territories, stated that the Agency could be more effective and foster greater engagement with other government departments. Similar sentiments were expressed by non-governmental organization stakeholders, international partners and academics.

Recognizing that there are many potential ways to achieve results in addressing community associated infections, the Centre for Communicable Diseases and Infection Control will need to focus on collaborative activities that further advance current Public Health Agency priorities.

RECOMMENDATION 1

Review the Public Health Agency’s collaborative activities in community associated infections.  Where gaps in achieving Public Health Agency outcomes are identified, develop an appropriate response to enhance coherence in prevention and control activities.

Knowledge translation

Community associated infections activities and outputs aim to achieve increased awareness on the state, trends, prevention and control public health practices for communicable disease in Canada. Direct outputs of Public Health Agency activities include but are not limited to: guidelines for health professionals and decision makers, professional desk references, surveillance reports, brochures and other public awareness products, training materials and tools, publications and research outputs.

In the past five years, the prevention work in the field of sexually transmitted infection conducted by Centre for Communicable Diseases and Infection Control has been mainly about knowledge development, translation and exchange. It appears as though the knowledge products produced by the Centre are appropriate. They appear to address the current and emerging needs, such as antimicrobial resistance. Survey respondents and key informants did not provide any indication that the Public Health Agency was not developing products to meet their needs.

However, partners and stakeholders noted that timeliness was a key obstacle in using the Public Health Agency’s products. The evaluation found that timeliness was impacting the Public Health Agency’s ability to be relevant in the Agency’s community associated infections role. 

While the Centre for Communicable Diseases and Infection Control has taken steps to ensure that some products are available to stakeholders in a more timely fashion, standards, protocols or procedures should be developed to ensure products are generally available when needed.

RECOMMENDATION 2

Review internal processes to improve the timeliness of community associated infection product distribution. Where needed, develop and implement standards to address identified gaps.

Performance measurement

The first step in a performance measurement study is to determine whether or not the program was implemented according to its proposed theory of change (typically represented by the program or activity’s logic model). The measurement and assessment of the Centre for Communicable Diseases and Infection Control’s progress towards the achievement of its intended outcomes appears to be inconsistent and unsystematic. As such, the Centre can speak to several outcome studies demonstrating success and areas for improvement, but is unable to frame its work into a comprehensive story of what it has achieved.

There has been systematic tracking of both outputs and outcomes of project work funded through the hepatitis C grants and contribution program. However, despite the national roll-out of this tool over three full fiscal years, the data was never rolled up into a national report, and there is no evidence that the information was used to inform program or policy decisions at the national level. There were a number of factors that contributed to this situation. Regardless, while the information was collected and available, it was never used to measure, assess and adjust performance for the overall hepatitis C program.

There are promising activities currently in the works which hint at the development of a more comprehensive performance measurement culture within the Centre for Communicable Diseases and Infection Control. The Centre is beginning to develop tools and methodologies to measure and assess performance. While these early developments are positive indications of a more effective performance measurement function with the Centre, future work must be comprehensive in its approach (documenting outputs and outcomes of both direct and grants and contributions activities), grounded in a clear theory of change, systematically integrated into operations, and then actively used in management practices to assess and adjust performance.

RECOMMENDATION 3

Develop a comprehensive and feasible performance measurement system to help support programmatic decision making.