Evaluation of the Federal Initiative to Address HIV/AIDS in Canada 2008-09 to 2012-13

Prepared by
Evaluation Directorate
Health Canada and the Public Health Agency of Canada

January 2014

Table of contents

• Executive summary
• Management Response Action Plan January 2014 Final
• 1. Evaluation Purpose
• 2. Program Description
  • 2.1 Program Context
  • 2.2 Program Profile (as of 2013)
  • 2.3 Program Logic Model and Narrative
  • 2.4 Program Resources
• 3. Evaluation Description
  • 3.1 Evaluation Scope
  • 3.2 Evaluation Issues
  • 3.3 Evaluation Approach and Design
  • 3.4 Data Collection and Analysis Methods
  • 3.5 Limitations and Mitigation Strategies
• 4. Findings
  • 4.1 Relevance: Issue #1 — Continued Need for the Program
  • 4.2 Relevance: Issue #2 — Alignment with Government Priorities
  • 4.3 Relevance: Issue #3 — Alignment with Federal Roles and Responsibilities
  • 4.4 Performance: Issue #4 — Achievement of Expected Outcomes (Effectiveness)
  • 4.5 Performance: Issue #5 — Demonstration of Economy and Efficiency
• 5. Evaluation conclusions and recommendations
  • 5.1 Relevance Conclusions
  • 5.2 Performance Conclusions
• 6. Recommendations
• Appendix 1: Logic Model
• Appendix 2: Federal Initiative Allocation and Expenditures, 2008-09 to 2012-13
• Appendix 3: Summary of Findings
• Appendix 4: Details on Data Collection and Analysis Methods
• Appendix 5: HIV Statistics
• Appendix 6: References
• List of Acronyms
• Endnotes

Executive summary

This evaluation covered the Federal Initiative to Address HIV/AIDS in Canada for the period of 2008-09 to 2012-13 with a corresponding average annual investment of approximately $73 million. The evaluation was undertaken in fulfillment of the requirements of the Financial Administration Act and the Treasury Board of Canada's Policy on Evaluation (2009).

Program Description

The Federal Initiative is a horizontal initiative involving four federal government organizations: the Public Health Agency of Canada (lead), CIHR, CSC and Health Canada. The Federal Initiative builds on the federal government's original Canadian Strategy for HIV/AIDS, a $42.2 million/year strategy starting in 1998. An additional $30.4 million annually is committed by the Federal Initiative for a total annual investment of $72.6 million. The Federal Initiative works towards the following four goals: to prevent the acquisition and transmission of new HIV/AIDS infections; to improve the quality of life for those at risk and living with HIV and AIDS; to contribute to the global effort to reduce the spread of HIV and AIDS and mitigate its impact; and to reduce the social and economic impact of HIV/AIDS.

Evaluation Purpose, Scope and Design

The purpose of the evaluation was to assess the relevance and performance of the Federal Initiative. The evaluation was also designed to support program planning and decision making. An implementation evaluation of the Federal Initiative was conducted in 2009. Recommendations in that evaluation concentrated on the development of performance measurement and on strengthening the program's governance. Activities conducted over the past five years addressed these recommendations.

This evaluation focused specifically on activities of the Federal Initiative. As such, the following activities were considered out of scope: activities related to tuberculosis, hepatitis B and C and sexually transmitted infections; the Canadian HIV Vaccine Initiative; and any efficiencies gained from integrating HIV/AIDS activities with other communicable disease areas conducted by all four departments (Health Canada, Correctional Service Canada (CSC), Canadian Institutes of Health Research (CIHR) and the Public Health Agency of Canada).

The methodology used in the evaluation included literature and document reviews and key informant interviews (54 in total). Assessments of individual activities in addition to ongoing performance measurement information provided evidence in assessing the performance of the Federal Initiative over the past five years.

Evaluation Conclusions and Recommendations

HIV/AIDS remains a public health issue for Canada that disproportionately affects vulnerable populations. Many needs persist almost a decade after the creation of the Federal Initiative. Prevention is still needed as new cases, while stable, continue to be diagnosed. The need for diagnosis, care, treatment and support has increased because people are living longer with the disease as it progresses to a chronic condition. HIV is costly; however, it is estimated that for each case of HIV that is prevented, an estimated $1.3 million can be saved.Footnote 1

The Federal Initiative aligns well with the mandate of each of the federal government partners. There is an appropriate role for the federal government in national and international leadership, coordination and knowledge translation for HIV/AIDS prevention, diagnosis, care, treatment, support and research. The Federal Initiative role aligns with the federal role of other countries such as Australia, the United Kingdom and the United States.

As the context for addressing HIV/AIDS has changed over the past decade, so has the Initiative. As such, many HIV/AIDS activities in the area of prevention, testing, diagnosis, care and treatment can target multiple infections at once, as opposed to addressing HIV/AIDS in isolation. Furthermore, there is an international movement towards integrating HIV/AIDS work within broader approaches that address areas of co-infection, common risk factors and common transmission routes. The current direction of integrating these types of activities should lead to efficiencies for partners and a broader approach to addressing the disease. Due to increased knowledge regarding similar risk factors and transmission routes, evidence shows that people at risk of contracting HIV are also at risk of contracting other sexually transmitted and bloodborne infections, such as hepatitis C, and are often co-infected.

The Federal Initiative recognizes that HIV/AIDS is a complex issue that requires a collaborative, multisectoral response that is informed by surveillance, research and community-based evidence. It assumes that increased knowledge and awareness of the nature of HIV/AIDS and ways to address the disease will help enhance individual and organizational capacity. These activities are expected to reduce stigma and discrimination associated with HIV/AIDS, which should lead to improved access to effective prevention, diagnosis, care, treatment and support. Ultimately, this should result in a lower infection rate and an improved quality of life for those living with the disease.

This is not a linear process and activities to address knowledge, for example, may also impact individual capacity, reduced stigma and improved access. There are also a number of activities, organizations and individuals, both internal and external to those conducted by Federal Initiative partners, that work towards the same goals.

The Federal Initiative is making solid progress in its efforts to address HIV/AIDS and there are many examples of success outlined in the body of the report. For example, there is evidence that Federal Initiative support has led to the creation of new and valuable knowledge which has helped to better understand and combat HIV/AIDS in Canada and abroad. At the same time, the support of the Federal Initiative has helped Canadian researchers to achieve a standing of international excellence. In addition, knowledge created through other Federal Initiative supported programs has led to many positive steps forward in programming and front line practice settings.

There is also evidence that individual and organizational capacity to address HIV/AIDS has been enhanced through Federal Initiative activities, especially in the context of technical support and community work. Furthermore, there are lots of individual examples of capacity building activities and there is evidence that an increase in capacity is occurring, especially in the research field. Federal Initiative partners have made substantial contributions in the global community, where partners deliver training and provide advice to organizations abroad, and participate in the development of joint documents to further advance the ability to address HIV/AIDS globally. Many collaborative activities have occurred, ensuring participation by targeted stakeholders.

There is evidence of impact of the Federal Initiative's activities. The report outlines examples of successful local or community-based projects to reduce stigma and discrimination, and there are also many examples of improved access to more effective prevention, diagnosis, care, treatment and support as a result of Federal Initiative activities. Finally, there is evidence that federal coherence in the approach to HIV/AIDS is strong in spite of a lack of joint work planning and priority-setting exercises.

As the Federal Initiative evolves, activities should be enhanced to maximise benefits to Canadians. To begin with, joint planning should be strengthened to ensure that activities are strategic and aligned with Government of Canada and departmental priorities. Secondly, given the extensive knowledge that has been developed on the disease and ways to address it, the Federal Initiative should strengthen efforts to share and apply knowledge (including research and surveillance) to policy and practice to advance public health action in this area. Thirdly, additional opportunities exist to address the barriers outlined in the original program authorities (e.g. poverty and lack of housing). Finally, partners in the Federal Initiative should further develop tracking mechanisms to monitor activities against work plans and make adjustments based on learnings. As the Federal Initiative is evolving, so should the logic model that explains the program, along with any related performance measurement strategy.

Recommendations have been developed to address the conclusions outlined above. However, while these recommendations are distinct, they are all interrelated. For example, addressing the first recommendation (planning and priority setting) should help focus activities to support how knowledge, such as research and surveillance, informs program decision making. This should inform how Federal Initiative partners can address barriers to prevention, diagnosis, care, treatment and support. These types of activities should be tracked and documented to ensure that learning is continuous.

Recommendation 1:

Improve joint work planning and priority setting

The Responsibility Centre Committee mechanism is currently used primarily for information sharing rather than decision making, strategic planning, and priority setting at the Federal Initiative level. Improved horizontal engagement would ensure ongoing discussions on both the short and long term strategic directions of the Federal Initiative as well as integrated advice to senior management on ways to address current and emerging HIV/AIDS needs in Canada.

In response to the 2009 evaluation of the Federal Initiative, a revised Terms of Reference for the Responsibility Centre Committee was developed to further enhance horizontal management. An annual review of an integrated work plan and identification of priorities, also identified as a deliverable in that evaluation, was not consistently conducted. The need for this type of exercise remains.

Recommendation 2:

Enhance knowledge exchange and application by supporting knowledge translation activities and opportunities to implement promising practices

There is evidence that knowledge translation and exchange is occurring and some practices, programs and policies have adapted based on the vast amount of knowledge developed through Federal Initiative activities. Key informants, both those working within the Federal Initiative and those external to it, agree that there has been a significant amount of knowledge and understanding that has been developed and disseminated over the years. While there is a recognition that new information will be needed to address emerging needs associated with HIV/AIDS, one of the challenges outlined in the evaluation was the ongoing need for federal management of enhanced knowledge translation activities.

1. Knowledge exchange: While a lot of knowledge and information has been produced and disseminated, the Federal Initiative should increase its efforts to foster a culture of sharing information and knowledge exchange.
2. Knowledge application: The Federal Initiative should strengthen efforts to apply knowledge (including research and surveillance) to policy and practice to advance public health action in this area.

The Federal Initiative could play a significant role in bringing partners together across sectors and ensure that intervention work has a vision and strategic plan, which may lead to increased engagement and increased coordination across Federal Initiative partners. As well, these efforts can provide opportunities for Federal Initiative partners and other stakeholders to connect and share information on HIV/AIDS activities to advance the Federal Initiative's efforts to address HIV/AIDS in Canada.

Recommendation 3:

Strengthen activities to address barriers to prevention, diagnosis, care, treatment and support

The 2004 program authorities outlined the need to address barriers to prevention, diagnosis, care, treatment and support. Evidence has shown that stigma and discrimination still exist and reducing those and other barriers will, in turn, improve access to prevention, diagnosis, care, treatment and support for those living with HIV/AIDS. Even though some work has been carried out in this area, further work should be done to address the social determinants of health as well as stigma and discrimination. There is a recognition that there are limitations to the work those in the health portfolio can accomplish as often, activities to address barriers are part of the mandate of other jurisdictions. Therefore, work in this area should focus on engaging with others who are responsible for key social determinants of health.

Recommendation 4:

Enhance internal collection and use of information

The Federal Initiative has advanced the collection of performance measurement over the last five years, partially as a response to the last evaluation. There was a lot of performance measurement information available to assess activities and outcomes for this evaluation. Therefore, the ability to assess success in terms of achieving outcomes was possible. However, improvements to performance measurement can enhance the ability to assess success on a continuous basis, including how knowledge generated is used to inform policy and practice.

1. Monitor and adjust for continuous improvement. While key informants noted anecdotal evidence of knowledge exchange, bilateral meetings, policy change, and international meetings, often this was not documented in a way that could be used to monitor activities and thus improve them on an ongoing systematic basis. Frequently, this falls outside traditional mechanisms of performance measurement. However, tracking these types of activities will help the Federal Initiative to continuously improve over the short and long-term.
2. Update the logic model and performance measurement strategy to reflect the evolution of the Federal Initiative. The Federal Initiative has evolved over the past five years based on the changing nature of HIV/AIDS as well as government direction in this area. The logic model and accompanying performance measurement strategy should be reviewed in light of these changes to ensure that activities and outcomes still align with the evolution of the Federal Initiative. This should include developing mechanisms of collecting information that focus on the impact of activities of the Federal Initiative in addition to what is produced.