Section 1: Evaluation of community associated infections prevention and control activities at the Public Health Agency of Canada – Introduction

1. Introduction

1.1 Purpose and scope of the evaluation

The purpose of the evaluation is to assess the relevance and performance of the community associated infections prevention and control activities at the Public Health Agency, including evaluative information needed to address Financial Administration Act requirements for review of activities to address hepatitis C in Canada. This evaluation should also assist senior management in program planning and decision making.

The evaluation addresses the relevance of these activities, including the continued need for this program and its alignment with federal government roles and priorities. It also addresses the performance of the Public Health Agency in achieving expected outcomes and demonstrating efficiency and economy. The time period examined is between April 2008 and October 2012.

The following questions guided the collection of data. All of these questions are covered by the evaluation. Key findings for each question were then synthesized into broader conclusions.

Relevance

• Continued need:
  • What is the magnitude of community associated infections in Canada?
  • Has this problem changed over time?
• Alignment with federal and agency roles and responsibilities:
  • In preventing and controlling community associated infections, what is the federal role?
  • What is the Public Health Agency’s current role in preventing and controlling community associated infections? How is the Public Health Agency’s current role aligned with federal role and responsibilities?
  • How is the Public Health Agency’s current role different from other government departments, provinces and territories, and non-governmental organizations? What are the gaps? Is the Public Health Agency doing what others should be doing?
• Alignment with government and Public Health Agency priorities:
  • Does our current role align with federal government and current Public Health Agency priorities?
• What is the Public Health Agency’s strategic approach to preventing and controlling community associated infections?

Performance

• Achievement of expected outcomes:
  • To what extent have the Public Health Agency’s activities resulted in increased collaboration with provinces, territories and stakeholders in the area of community associated infections?
  • To what extent have activities that are conducted and/or supported by the Public Health Agency helped to increase awareness and knowledge about community associated infections?
  • What evidence exists on the risks and impact of community associated infections? To what extent is evidence used by provinces, territories and stakeholders to develop/improve policies and programs?
• Demonstration of Efficiency and Economy:
  • Is there variance between planned and actual expenditures? If so, what are the implications?
  • Are community associated activities achieving their intended outcomes in the most economical manner?
  • Are activities undertaken and products delivered in the most efficient way?
  • To what extent is there performance measurement in place? How is the program area using performance measurement to improve the design and delivery of activities? How could the efficiency of the activities be improved? Are there alternative, more efficient ways of delivering the activities?

This report describes what the Public Health Agency does to prevent and control tuberculosis, hepatitis B and C, sexually transmitted infections, including the promotion of sexual health. It assesses links, outlines challenges and opportunities, and culminates in key findings and conclusions which in turn lead to three recommendations.

The scope of this evaluation does not include the Federal Initiative to Address HIV/AIDS in Canada. An evaluation of this initiative will take place separately as will an evaluation of the Public Health Agency’s activities for the Canadian HIV Vaccine Initiative. Similarly, the Public Health Agency’s activities in travel and border health, health-care associated infections as well as pandemic and avian activities conducted by the Centre for Communicable Diseases and Infection Control will also be covered under separate Public Health Agency evaluations.

Hepatitis C Undertaking Initiative

The Hepatitis C Undertaking Initiative provides payments to provinces and territories to improve access to health care and treatment services to persons infected with hepatitis C through the blood system prior to 1986 and after 1990. This federal initiative provides $300 million in transfer payments over a 20-year period to assist the provinces and territories in the provision of hepatitis C health care services.

In 2009, the Public Health Agency conducted an evaluation of the Hepatitis C Undertaking Initiative. The evaluation report concluded that the Initiative addressed the demonstrable need and that there was a link between the objectives of the initiative and federal government priorities as well as the Agency's strategic outcome. The evaluation found that the federal government fulfilled its responsibilities according to the Initiative. The fundamentals of the Initiative have not changed and no further funding has been provided to the provinces and territories since 2009. Therefore, there is no need to evaluate the Initiative through this evaluation.

1.2 Methodology

Approach and design

This evaluation was conducted by the Evaluation Directorate, a group internal to the Public Health Agency but not involved in the program areas responsible for the administration of community associated infections activities.

This evaluation incorporates multiple lines of evidence and a combination of qualitative and quantitative measures to ensure a balanced analysis of the relevance and performance of community associated infections activities.

Methods

This evaluation used multiple lines of evidence, including a review of literature and relevant documents, as well as key informant interviews and surveys. Performance measurement collected to report on activities associated with the Hepatitis C Prevention, Support and Research Program’s grants and contributions was also used as a line of evidence for this evaluation.

The evaluation matrix, which outlines evaluation questions and lines of evidence, can be found in a technical report, available upon request. Evaluators conducted the interviews using structured interview guides that followed the evaluation questions identified in the evaluation framework. Evaluation tools, such as interview guides, are also available in the technical report.

1. Literature review

Evaluators conducted a review of literature to explore the magnitude of community associated infections in Canada and whether the incidence/prevalence has changed over time. The purpose of this review was to obtain information about whether there was a continued need for the Public Health Agency’s current activities in preventing and controlling infections such as tuberculosis, hepatitis C and other sexually transmitted infections such as syphilis.

The 67 documents reviewed included literature (published and unpublished) that Centre for Communicable Diseases and Infection Control program staff suggested, along with documents identified through Internet searches. As well, database searches were conducted in Medline, Embase, Global Health and EconLit for English-only articles published after 2007. The search focused on the articles on antimicrobial resistance in community associated infections, hepatitis B and C, sexually transmitted infections and tuberculosis.

2. Program file and document review

The Evaluation Directorate reviewed the program files and documents available on community associated infections activities at the Public Health Agency. Files and documents were initially identified by Centre for Communicable Diseases and Infection Control program staff and through a search of relevant websites. This review addressed the evaluation issues related to relevance and performance. In total, evaluators reviewed 109 documents, not including websites. The types of documents reviewed included:

• program authorities, acts, protocols and agreements
• performance measurement information (where available)
• surveillance reports
• strategic plans
• external evaluation reports
• foundational documents
• briefing notes and decks
• public opinion research
• financial data.

3. Interviews with key Public Health Agency senior managers, staff in the area of community associated infections, as well as other key external stakeholders

The interviews addressed evaluation issues related to relevance and performance, including roles and responsibilities, the description of community associated infections activities at the Public Health Agency and their performance. The interviews also identified areas for improvement and existing gaps (challenges).

Individuals with involvement or experience in community associated infections, either within the Public Health Agency or externally, were targeted for the interviews. Following an initial scoping interview with key program contacts, additional names, both internal and external to the Public Health Agency, were suggested for an interview. From July to November 2012, 34 one-hour (on average) interviews were conducted. These interviews were audio-recorded. They were transcribed (verbatim) and used for analysis.

The breakdown of interviewees by type of key informant is as follows.

Surveys

The Evaluation Directorate also conducted four on-line surveys with various groups to ask them about the Public Health Agency’s roles and activities in community associated infections. Four separate surveys were undertaken.

• The federal/provincial/territorial Public Health Network Communicable and Infectious Disease Steering Committee: The survey was sent to 14 potential respondents, all members of the Steering Committee who were external to the Public Health Agency. Committee members were invited to forward the survey to other potential respondents within their jurisdiction, if appropriate. Seventeen respondents completed the online questionnaire. (A response rate is inappropriate as a snowball approach was employed. Due to the small and focussed sample, percentages will not be used to present results.)

The following three surveys employed a purposive sampling technique. After initial scoping interviews with key program contacts, names of stakeholders external to the Public Health Agency were provided for the survey samples. Often, these were lists of people who had been engaged by the Agency on working or technical groups or had requested information from the Agency on their products.

The surveys were conducted in a staggered approach between September 29 and November 2, 2012 and were available (open) for at least three weeks. Each sample list was confirmed by the Centre for Communicable Diseases and Infection Control.

• The non-profit sector: The survey was sent to 145 potential respondents, with a request to forward on the survey to other potential respondents within their jurisdiction, if appropriate. Forty-one respondents completed the online questionnaire.
• Researchers in the field: The survey was sent to 45 potential respondents. Twenty-one respondents completed the online questionnaire.
• Other federal government departments: The survey was sent to 36 potential respondents, with a request to forward on the survey to other potential respondents within their jurisdiction, if appropriate. Nineteen respondents completed the online questionnaire.

Fluid Surveys software was used to administer the survey and support data analyses.

Additional information on data collection techniques is outlined in the technical report which is available upon request. The Evaluation of Community Associated Infections Activities at the Public Health Agency: Technical Report contains the questionnaires and a more detailed breakdown of survey results.

Limitations

Most evaluations face constraints that may have implications for the validity and reliability of evaluation findings, conclusions and recommendations. This section illustrates the limitations in the design and methods for this particular evaluation. Also noted are the mitigation strategies put in place by the evaluation team to ensure that the evaluation findings can be used with confidence to guide program planning and decision making.