Evaluation of the Canadian Agency for Drugs and Technologies in Health Activities 2012-2013 to 2015-2016

List of Tables

Table 1: Program resources ($)

Table 2: Limitations and mitigation strategies

Table 3: CADTH revenue, 2012-2013 to 2015-2016 ($)

Table 4: Variance between planned spending and actual expenditures for CADTH, 2013-2014 and 2015-2016 ($)

Table 5: CADTH's planned and actual production by major line of products, 2012-2016

Table 6: CADTH's outputs by jurisdiction, 2012-2016

Table 7: CADTH datasets used in this evaluation

Table 8: Distribution of interviewees by group

List of Acronyms

Executive Summary

This evaluation covered the activities of the Canadian Agency for Drugs and Technologies in Health (CADTH) for the period from 2012-2013 to 2015-2016. The evaluation was undertaken in fulfillment of the requirements of the Financial Administration Act and the Treasury Board of Canada's Policy on Results (2016).

Evaluation Purpose and Scope

The purpose of the evaluation was to assess the relevance and performance of CADTH's activities. The scope of the evaluation covered various CADTH activities from April 2012 to March 2016, including health technology assessments (HTAs), optimal use projects and common drug reviews (CDR). Where warranted, data outside this timeframe was included in the evaluation.

The evaluation did not cover the pan-Canadian oncology drug review (pCODR) program itself as it was transferred from the provinces and territories (P/Ts) to CADTH during this evaluation; however, the transfer process was examined.

CADTH was last evaluated in 2014 by the Office of Audit and Evaluation as part of the Synthesis Evaluation of Transfer Payments to Pan-Canadian Organizations 2008-2009 to 2012-2013. In addition, the most recent recipient-led evaluations were conducted in 2012 and 2016.
As outlined in section 3.1, the consulting firm of Science-Metrix conducted the recipient-led evaluation of CADTH's activities.

Program Description

CADTH is an independent, pan-Canadian organization created in 1989 by the Federal, Provincial and Territorial (F/P/T) Ministers of Health to support the optimal use of drugs and non-drug technologies in Canada's health care system. CADTH seeks to provide timely, evidence-based information about the clinical and cost-effectiveness of pharmaceuticals and other health technologies (including devices, procedures and systems).

CADTH's customers are decision makers (policy, practice, procurement) in ministries and departments of Health and publicly funded organizations responsible for health service delivery, such as health authorities, health facilities (e.g., hospitals, long-term care facilities) and public health agencies (e.g., cancer agencies, transplant agencies, centres for disease control). As Quebec is not a CADTH funder, the Ministry of Health and the health regions, hospitals, and other groups funded by that provincial health ministry are not considered customers. Similarly, due to the funding arrangement with Ontario, the Ontario Ministry of Health and Long-term Care is a customer for drug reviews, but it (and the local health integration networks, hospitals, long-term care facilities, and other groups funded by the provincial health ministry) is not a direct customer for HTA work. CADTH's reports are freely available to any interested party on the CADTH website.

Conclusions - Relevance

Continued Need

There is a continued need for CADTH's assessments of drugs and health technologies as these are a source of information in evidence-based decision making. CADTH activities are important given the increased demand for health technology assessments resulting from innovation in health technologies and convergence of drug and non-drug technologies.

Alignment with Government Priorities

As outlined in a variety of strategic documents, protecting the health and safety of Canadians remains a priority for the Government of Canada. The Government funds pan-Canadian organizations such as CADTH to support health system innovation and sustainability which also remain priorities for the Government as evidenced by their inclusion in recent federal agenda-setting documents such as the Speeches from the Throne, Budgets and the 2015 Minister of Health mandate letter.

Alignment with Federal Roles and Responsibilities

Health Canada has a clear federal role to act as a catalyst for innovation, influencing health care policy and providing evidence-based information to health care decision makers. The activities carried out by CADTH are aligned with the Minister's legislative authorities under the Department of Health Act (1996).

Health Canada makes investments to help the P/Ts accelerate change in areas of shared priority. Health Canada supports CADTH's mandate to deliver reliable, timely, evidence-based information to Canada's health care leaders on the effectiveness and efficiency of health technologies. This information is particularly useful to P/Ts which do not have the capacity to carry out their own assessments.

CADTH's role as both a producer of evidence and a knowledge mobilizer in health technologies (both drug and non drug) is complementary to the role of the Canadian Institutes of Health Research's Drug Safety and Effectiveness Network (DSEN). CADTH produces evidence and performs health technology assessments using the current available research. It also acts as a knowledge mobilizer to translate and exchange that evidence with decision makers to inform policy and practice. Meanwhile, DSEN supports research and researcher capacity in the area of drug safety and effectiveness, it does not carry out research on non-drug technologies, nor does it assess the cost-effectiveness of drugs. In addition, DSEN research is often used by CADTH to inform HTA processes.

Conclusions - Performance

Achievement of Expected Outcomes (Effectiveness)

While CADTH has made positive contributions to building awareness and understanding of health technology assessment and optimal use evidence, awareness of some of its new and traditional products and services was not consistently high across stakeholder groups. For example, the role and potential value of knowledge mobilization activities are well understood among external stakeholders, although the value of the knowledge mobilization function and the liaison officers' activities as a link between CADTH and its customer base were not always well understood internally. While CADTH has knowledge products that are readily accessible which has helped enhance its credibility, some groups have identified a need for additional transparency around the therapeutic and device review discussions.

CADTH helped increase awareness and understanding of its HTA and OU evidence. In addition, there was strong evidence that CADTH's products and services are being used by stakeholders. Users^Footnotea of CADTH's products and services were highly satisfied with their quality, utility, relevance and credibility. There were differences in the way customers used CADTH's products depending on their capacities to conduct HTA or to implement recommendations. Jurisdictions with limited capacity to carry out assessments have relied more heavily on CADTH's advice and recommendations to make decisions on coverage. CADTH has improved timeliness in responding to customers' requests, but it could have ensured better use of its products if information were available when listing and price negotiations were being made.

In addition to its stated outcomes, evidence shows that CADTH is one of the few HTA organizations (European network for Health Technology Assessment, National Institute for Health and Care Excellence, Scottish Medicines Consortium) that has been proactive in the systematic implementation of patient engagement as part of its HTA processes. Incorporating the patient perspective lends additional credibility to CADTH's products and services and increases the likelihood of uptake by customers.

In line with F/P/T governments' commitments to HTA collaboration and their goal of improving the efficiency and cost-effectiveness of public health care service provision, CADTH has introduced initiatives to improve coordination and knowledge sharing between HTA producers, and has changed its traditional relationships with some customers. Further, by fostering collaboration among health stakeholders, CADTH has contributed to reducing duplication in HTA processes in Canada. CADTH has also improved collaboration with most of the stakeholders, particularly those that have long-standing HTA capacities (e.g., Health Quality Ontario and Institut national d'excellence en santé et en services sociaux).

Demonstration of Economy and Efficiency

CADTH is primarily funded by the federal, provincial and territorial governments. The Government of Canada committed to providing CADTH with up to $80.6 million from 2013 to 2018 (of which $65.4 million was provided over the four years covered by this evaluation). Federal government funding to CADTH varied from 58% to 81% of CADTH's total annual funding from April 2012 to March 2016. In addition to federal funding, CADTH receives P/T funding as well as other funding that includes revenue from symposiums and workshops, interest revenue, other service revenue for work done for other organizations and industry application fees for the CDR program.

Prior to, and over the timeframe of the evaluation, CADTH continued to make improvements to the efficiency and effectiveness of the governance structure and practices (e.g., changes made to the composition of the Board of Directors and reporting changes); however, there are opportunities for further enhancements.

With respect to performance measurement, data collection efforts at CADTH are most successful at the activity and output levels, and while systems are in place to track the extent of CADTH's contribution to long-term outcomes (e.g., improved health outcomes, policy changes and policy coordination), this data is not readily obtained from the health system. These outcomes could not be assessed due to an absence of any secondary data and lack of informed stakeholder opinion.

Recommendations

The recipient-led evaluation resulted in three recommendations directed at CADTH which can be found on the CADTH website, whereas this departmental-led evaluation did not result in any recommendations directed at the department.

1.0 Evaluation Purpose

The purpose of the evaluation was to assess the relevance and performance of CADTH’s activities for the period of 2012-2013 to 2015-2016. However, where warranted, data outside this timeframe was included in the evaluation.

2.0 Program Description

2.1 Program Context

The Government of Canada's objective in supporting CADTH is to address the need to increase the access to, and use of, relevant evidence to inform the optimal and cost-effective use of drugs and health technologies. In so doing, the Government of Canada seeks to harness the benefits of health technologies while getting the best value from its investments in health. As such, CADTH's mandate and objectives are consistent with Health Canada's Program Alignment Architecture, which links Health Canada's strategic outcome of "A Health System Responsive to the Needs of Canadians," to the program of "Canadian Health System Policy".

CADTH was last evaluated in 2014 by the Office of Audit and Evaluation as part of the Pan-Canadian Synthesis evaluation. In addition, the most recent recipient-led evaluations were conducted in 2012 and 2016.

2.2 CADTH Profile

CADTH is an independent, pan-Canadian organization created in 1989 by the F/P/T Ministers of Health to support the optimal use of drugs and non-drug technologies in Canada's health care system. CADTH seeks to provide timely, evidence-based information about the clinical and cost-effectiveness of pharmaceuticals and other health technologies (including devices, procedures and systems).

Throughout this document, the term device is intended to encompass medical devices; diagnostic tests; and medical, surgical, or dental procedures and programs. CADTH works closely with its customers (i.e., health decision makers) to produce "unbiased information and advice, using the best available evidence. CADTH acts as a catalyst, collaborating with other Canadian and international HTA producers and experts, to broker knowledge and leverage the health technology assessment capacity and resources available."^Footnote1

Similarly, CADTH conducts health technology assessments (HTAs) and assesses the optimal use of products for "clinical effectiveness and/or cost-effectiveness, and may include the ethical, legal, and social implications of health technologies on patient health and the health care system."^Footnote2 Health technology management (HTM) is a broader term that encompasses different aspects of the planning, and use and management of health technology assets in health care organizations (see Appendix 1 for more information on CADTH's products and services).

Customers and Key Stakeholders

CADTH's customers are decision makers (policy, practice, procurement) in ministries and departments of Health and publicly funded organizations responsible for health service delivery, such as health authorities, health facilities (e.g., hospitals, long-term care facilities) and public health agencies (e.g., cancer agencies, transplant agencies, centres for disease control). As Quebec is not a CADTH funder, the Ministry of Health and the health regions, hospitals, and other groups funded by that provincial health ministry are not considered customers. Similarly, due to the funding arrangement with Ontario, the Ontario Ministry of Health and Long-term Care is a customer for drug reviews, but it (and the local health integration networks, hospitals, long-term care facilities, and other groups funded by the provincial health ministry) is not a direct customer for HTA work. CADTH's reports are freely available to any interested party on the CADTH website.

CADTH stakeholders include clinicians, patient groups, provincial and national health associations, other HTA producers (both within Canada and internationally), academic institutions, partner organizations such as other pan-Canadian health organizations (e.g., Canadian Partnership Against Cancer, Canadian Patient Safety Institute, Canadian Institute for Health Information, and Canadian Foundation for Health care Improvement) and the federal departments responsible for certain populations (i.e., First Nations and Inuit, armed forces, veterans and inmates).

2.3 Program Narrative

The focus of this evaluation is on the CADTH activities funded by Health Canada. CADTH's various Health Canada funded activities include HTA, optimal use and (CDR). These serve to advance CADTH's role as a producer and broker of evidence. Combined, these activities form the basis upon which CADTH's performance is assessed.

The CADTH logic model was developed in collaboration with Health Canada. It covers CADTH activities, with the exception of the pCODR and the Scientific Advice Program, and illustrates the links between CADTH's activities and its intended outcomes (Appendix 2). CADTH uses the logic model to guide its program cycle from program design and planning, through implementation, to the final program evaluation and strategic reporting.

The theory of change that underlies CADTH's activities is to enable informed decision-making about health technologies by providing evidence-based information to health decision makers. Providing decision makers with timely evidence-based information that is relevant, of high quality, credible and perceived as independent, is intended to contribute to the optimal use of drug and other health technologies.

According to the program logic model, CADTH activities include: HTA production, generation of advice through expert committees, relationship maintenance in order to broker HTA knowledge, and reviews of submitted drugs that assess their clinical and cost-effectiveness. These activities are expected to lead to the following immediate, intermediate and long-term outcomes.

Immediate Outcomes

It is anticipated that program activities will increase: awareness and understanding of HTA and optimal use evidence, awareness of evidence for listing decision, and transparency across jurisdictions.

Intermediate Outcome

Achieving immediate outcomes is expected to: increase utilization of evidence-based information in health care decision making, improve coordination of drug and other health technology reviews, and improve collaboration among health care system stakeholders, including other producers and users of evidence.

Long-term Outcome

CADTH anticipates that achieving the above outcomes will lead to HTAs that inform every health technology decision.

2.4 Program Alignment and Resources

The program's financial data for the years 2012-2013 through 2015-2016 are presented below (Table 1). CADTH is funded by the Government of Canada, along with the provinces and territories. The Government of Canada provides the majority of CADTH's funding and committed to providing CADTH with up to $80,631,924 in funds to support CADTH's work as outlined in the Contribution Agreement effective April 1, 2013 to March 31, 2018.^Footnote3 The percentage of the federal contribution to overall revenue has decreased due to a relatively small reduction in funding from the federal government of 5% phased in over 2012-2013 and 2013-2014; one-time federal funding for work on isotopes ending in 2011-2012; and the introduction of industry application fees in 2014-2015. The Government of Canada provided $65.4 million to CADTH over the four years covered by this evaluation (see section 4.5 for further details).

3.0 Evaluation Description

3.1 Evaluation Scope, Approach and Design

The scope of the evaluation covered the period from April 2012 to March 2016 and included various activities such as HTA, OU and CDR. The evaluation did not cover the pCODR program itself as it was transferred from the P/Ts to CADTH during the evaluation; however, the transfer process was examined.

The evaluation issues were aligned with the Treasury Board of Canada’s Policy on Results (2016) and considered the five core issues under the two themes of relevance and performance, as shown in Appendix 3.

A theory-based evaluation approach was used for the conduct of the evaluation to assess the use of an explicit theory of change to draw conclusions about whether and how an intervention contributed to observed outcomes.

The Public Health Agency of Canada and Health Canada Office of Audit and Evaluation acknowledges and thanks all individuals who gave their time and input for this evaluation. In particular, the Office of Audit and Evaluation acknowledges the consulting firm of Science-Metrix which conducted the recipient-led evaluation of CADTH’s activities.

Data for the evaluation was collected using various methods, including document and literature reviews, administrative and financial data reviews, key informant interviews, an e-survey of rapid response service customers, and case studies. More information on the data collection and analysis methods is detailed in Appendix 3. In addition, data were analyzed by triangulating information gathered from the different methods listed above. The use of multiple lines of evidence and triangulation were intended to increase the reliability and credibility of the evaluation findings and conclusions.

3.2 Limitations and Mitigation Strategies

Most evaluations face constraints that may have implications for the validity and reliability of evaluation findings and conclusions. The following table outlines the limitations encountered during the implementation of the selected methods for this evaluation. Also noted are the mitigation strategies put in place to ensure that the evaluation findings can be used with confidence to guide program planning and decision making.

4.0 Findings

4.1 Relevance: Issue #1 - Continued Need for the Program

Assessments of drugs and other health technologies are an important source of information in evidence-based decision making. With an increased demand for Health Technology Assessments resulting from a combination of factors, including rapid innovation in health technologies and convergence in drug and non-drug technologies, there continues to be a need for CADTH activities in this area.

The International Network of Agencies for Health Technology Assessment defines health technology as an intervention that may be used to promote health, to prevent, diagnose or treat acute or chronic disease, or for rehabilitation.^Footnote4 Health technologies include a broad range of medical equipment/devices, pharmaceuticals, procedures and organizational systems used in health care.^Footnote5,^Footnote6 HTA is an evidence-based policy tool that helps inform decision-making on how to balance demand and supply pressures for new and existing technologies within a health system budget. HTAs assist in the development of strategies to enhance potential health gains within overarching fiscal constraints; it helps decision making to be a clear, transparent and coordinated process that incorporates economic evaluation and available evidence.^Footnote7

There has been increased demand for CADTH's assessments. For example, based on CADTH's annual business plans, requests for two of CADTH's main products/services have increased over the years covered by this evaluation.

• Rapid response service^Footnoteb submissions have increased 48%, from 236 in 2012 to 349 in 2015.
• CDR applications have increased 49%, from 33 in 2012 to 49 in 2016.

This increase in demand has resulted from a number of factors, including innovation in health technologies and convergence of drug and non-drug technologies. According to external key informants, there has been an expansion in the number of new drugs, especially for oncology and rare disease treatments meaning that there is an increased need for CADTH products and services. The new drug pipeline^Footnotec by pharmaceutical companies is expected to result in a significant number of new drugs launched to market in the coming years. Pharmaceutical firms are changing their business and research and development models. They are moving away from blockbuster drugs to niche areas in rare diseases and customized therapies, which are expected to have higher prices. It is likely that this would increase the importance of HTA on informing listing decisions. As well, it may be a factor if CADTH is to implement a priority review process or align its HTA with the regulators approvals or on priority process for drug reviews.

After a review of submissions to the CDR over the time period of this evaluation, it was noted that the volume of new drugs has increased the pressure on CADTH to complete more drug reviews within a given time frame. It has also required an expansion in the scope of CADTH's products and services to accommodate the growing need for information about drugs and non-drug health technologies. Customers (i.e., health care decision makers) are increasingly aware of the need to assess the value of health technologies to the health system as part of an informed decision-making process, and CADTH has responded by improving the links between HTA activities and customer priorities. Furthermore, there is a continued need for the relevant, credible evidence-based information that CADTH provides to F/P/T drug plan managers (except Quebec) about the optimal use of drugs and other health technologies.

Key informants noted that innovation in non-drug technologies is changing the way health systems replace their equipment. Replacement is increasingly due to technology becoming obsolete, as opposed to equipment coming to the end of its usable life. Furthermore, combining drug and non-drug technologies (e.g., drug-eluting stents^Footnoted and companion diagnostics^Footnotee) is an increasing trend. This trend implies more complex treatments, a more complex multi-stakeholder environment, and a systemic approach to health technology assessments in which health technologies are assessed in terms of their individual merits as well as in light of their possible interaction with other health technologies. All of which demonstrates the need for CADTH activities in this area.

Interviewees and survey respondents viewed CADTH as a valuable resource in the health care system, particularly for health care decision makers who need to respond to significant pressures and demonstrate a more rational use of scarce public resources. For example, drug plan managers indicated that they have had to respond to many and often disconnected demands from industry, patient groups, physicians, and government organizations; moreover, they need to respond in a way that maintains transparency and accountability in the use of public resources. Interviews and the literature review showed that decision makers find themselves needing more evidence to support their decisions on issues affecting health care provision, including financial constraints, demographic changes, and the costs of new technologies.

Interviewees also highlighted significant differences in information needs across jurisdictions. For example, the population of Atlantic Canada is, on average, older than in other parts of the country. As the population ages, the burden of chronic illness will rise, thus creating a greater need than in other regions for information on new developments in the treatment of chronic and age-related diseases.

4.2 Relevance: Issue #2 - Alignment with Government Priorities

Protecting the health and safety of Canadians is a priority of the Government of Canada. The Government funds pan-Canadian organizations such as CADTH to support health system innovation and sustainability, which remain priorities for the Government as evidenced by its inclusion in recent federal agenda-setting documents.

Government of Canada priorities

Improving the health of Canadians through health system innovation has been a longstanding priority for the Government of Canada. In 1989, the F/P/T Ministers of Health recognized the importance of a coordinated approach to evaluating technologies by creating CADTH to provide evidence-based information on emerging and existing medical devices. Its mandate has since expanded to include formulary reviews, optimal use projects and health technology assessments.^Footnote8

More recently, the Government of Canada has indicated through Speeches from the Throne, speech from the Minister of Health, federal budgets and the Minister of Health's mandate letter that the health and safety of Canadians and their families is a priority which is aligned with CADTH's goal to contribute to the effectiveness, quality and sustainability of health care in Canada.^Footnote9

In 2015, the mandate letter to the Minister of Health specifically identified advancing pan-Canadian collaboration on health innovation to improve outcomes for patients as a priority. The mandate letter also made reference to improving access to prescription medications through various means including exploring the need for a national drug formulary.^Footnote10,^Footnotef

The 2015 Speech from the Throne and the 2016 Budget both indicated that the government supports the health and well-being of all Canadians, and that the Government would work with the P/Ts to develop a new Health Accord^Footnote11,^Footnote12 that will improve health care in Canada and boost health outcomes for all Canadians. CADTH's activities in the areas of HTAs, OUs and CDR work towards improving health outcomes for Canadians.

A recent speech (September 2016) made by the Minister of Health confirmed that discussions are underway with the P/Ts and to date, there has been agreement on two priorities that are relevant for CADTH's work: enhancing the accessibility of prescription drugs and supporting pan-Canadian innovation. In her keynote address to the 2016 CADTH Symposium (April 2016), the Minister of Health stated that the government shared, "your goals of ensuring that decision making is evidence-based…" further showing support for CADTH's work.^Footnote13

Health Canada priorities

One of Health Canada's key priorities is supporting health system innovation. This priority has been highlighted in Health Canada's Reports on Plans and Priorities (2013, 2014 and 2015).

• Promote Health System Innovation by:
  • Working with P/Ts and other health care partners on health system renewal, innovation and sustainability; and
  • Addressing priority health issues through collaboration with key pan-Canadian organizations (such as CADTH), and the management of contribution programs and grants.

Alignment between CADTH activities and Health Canada and Government of Canada priorities

Health Canada funds CADTH to support the organization achieve its goal of producing and sharing evidence-informed information on drug and non-drug technologies in health to decision makers and supporting the adoption of these technologies, thereby contributing to health system renewal and sustainability.

Health Canada's recent Reports on Plans and Priorities highlight implementing innovative approaches to meet the health services needs of Canadians through organizations such as CADTH as a key priority. CADTH strategic plans note that the organization aims to enhance the health of Canadians by promoting the optimal use of drugs and other health technologies by producing advice, recommendations and tools and fostering evidence generation and adoption across Canada.^Footnote14,^Footnote15 The 2016-2017 Report on Plans and Priorities notes that CADTH evidence supports policy and practice decision making that can contribute to overall good health. Further, Health Canada's First Nations and Inuit Health Branch makes frequent Rapid Response Service requests and uses listing recommendations. The program has also cited CADTH evidence on its website.^Footnote16

The above mentioned CADTH plans are aligned with Health Canada's strategic priority: promoting health system innovation to improve the health of Canadians, which is also aligned with the Government of Canada priorities in this area.

Looking forward, as demand for CADTH reviews has increased, so has the expectation that CADTH would introduce holistic mechanisms for priority setting. The "first come, first served" practice may no longer meet the needs of decision makers.

Interview evidence showed that CADTH implemented changes to priority setting towards the end of the evaluation period. Initiatives included the following:

• Proactive identification of device-related topics that might have a significant impact on jurisdictions through informal discussions between CADTH liaison officers and health decision makers, and through an open call on CADTH's website. With medical devices, the list of topics is examined periodically according to a scoring tool, which has been available on the website since November 2015.^Footnote17 Topics are then ranked and validated (with respect to capacity and priority) by CADTH senior managers, liaison officers and the Health Technology Expert Review Panel. Customer interest and commitment are key criteria.
• CADTH has been mapping each HTA producer's work on devices and identifying sources of potential duplication. This work has been accomplished through two committees: the Health Technology Analysis Exchange, involving 12 jurisdictions in Canada (regional or hospital based) and aimed at sharing information on methods, practices and challenges; and the pan-Canadian Health Technology Assessment Collaborative, involving the four major HTA producers across Canada (CADTH, Institut national d'excellence en santé et en services sociaux, Institute of Health Economics, Health Quality Ontario) and aimed at identifying common priority areas.
• Priority review criteria were implemented on April 23, 2014 as a temporary measure as a result of the Common Drug Review backlog.

This evaluation found a perception that because of the traditional way in which CADTH approaches its pipeline-on a first-come, first-served basis-the order in which reviews are performed has been supply driven^Footnoteg for CDR and customer driven^Footnoteh for OU. According to interviewees, this practice of "first come, first served" as the basis for planning production of drug reviews should be replaced by a more proactive policy whereby CADTH sets some priorities on drugs to be reviewed-in consultation with provincial jurisdictions, for example. Prioritization was found to go hand in hand with the need for harmonization in the assessment of health technologies in general.

Interviewees recommended revisiting the type of reviews performed on biosimilar^Footnotei and rare disease drugs. They felt that the general framework applied to assess drugs was not adapted to biosimilar and rare disease drugs for various reasons. In regard to biosimilar drugs, given their similarities with other already assessed drugs, the time and level of effort expended could be reduced in order to reallocate resources to more challenging assessments. In the case of rare disease drugs, given the small population and thus the very limited evidence at the time of regulatory approval, the assessment has a limited added-value when undertaken. Instead, the monitoring of such items after their introduction on the market and their later evaluation when data are more available, as well as a coordination and information sharing with other international HTAs, may be more helpful.

4.3 Relevance: Issue #3 - Alignment with Federal Roles and Responsibilities

Health Canada is a catalyst for innovation, influencing health care policy and providing evidence-based information to health care decision makers. The activities carried out by CADTH are consistent with federal roles under the Department of Health Act and program authorities.

Health Canada's mandate includes the promotion and preservation of the health of Canadians. This includes its role as a catalyst for innovation, a funder of grants and contributions to meet overall health system objectives, as well as, a provider of information enabling health care decision makers and Canadians to make informed decisions.^Footnote18 While the provinces and territories are responsible for delivering health care to the majority of Canadians, the federal government also has a number of key roles and responsibilities in areas that affect health:

• promoting and preserving the physical, mental and social well-being of the people of Canada;
• protecting Canadians against risks to health;
• establishing and controlling safety standards and safety information requirements for consumer products;
• collecting, analysing, interpreting, publishing and distributing information related to public health;
• cooperating with provincial authorities with a view to the coordination of efforts for preserving and improving public health; and
• providing a limited range of medically-necessary, health-related goods and services to eligible First Nations and recognized Inuit when not otherwise provided through other public programs or private insurance plans.^Footnote19,^Footnote20

As a producer of HTAs, drug reimbursement recommendations and a broker of evidence, CADTH produces and shares evidence-informed knowledge on drugs and non-drug technologies in health with decision makers to support the adoption of these technologies which can directly enhance the health and safety of patients across Canada.

Before a drug can be sold in this country, Health Canada must receive evidence demonstrating the safety, efficacy and quality of the product. Once approved, the drug can be prescribed, but it may or may not be reimbursed by individual P/T drug plans. Prior to being added to the F/P/T drug plans (except Quebec); drugs undergo a clinical and cost-effectiveness assessment by CADTH. Each publicly funded drug plan and cancer agency (except Quebec) considers the recommendations of the CADTH review in addition to other factors (e.g., budgets) before making a decision on coverage.^Footnote21

Similarly, before a medical device can be sold in this country, Health Canada must receive evidence demonstrating the safety, efficacy and quality of the product. Once approved for market, there is no central process for health technology assessments. British Columbia, Alberta, Ontario, Quebec and Newfoundland have developed their own provincial processes, whereas CADTH undertakes health technology assessments deemed to be of national interest at the request of P/T governments. A recent federal government Advisory Panel report noted that this service is particularly helpful for those provinces which do not have their own capacity to carry out such reviews.^Footnote22 Further, according to key informants, the larger jurisdictions that have their own in-house capacities have recognized CADTH's contribution in their own decision-making process and have started to collaborate with CADTH to avoid duplication.

Health Canada influences health care policy across Canada through pan-Canadian organizations such as CADTH. The federal government makes investments to help the provinces and territories accelerate change in areas of shared priority. As such, Health Canada supports CADTH's mandate to deliver reliable, timely, evidence-based information to Canada's health care leaders on the effectiveness and efficiency of health technologies. Products and services available through CADTH are used by decision makers within the ministries of health and the federal government for certain populations (for example, FNIHB, Corrections Service Canada) to support decisions on which medical devices and drugs to adopt and use. Such decisions are important to the quality, affordability and sustainability of health care in Canada while getting the best value from every health care dollar.^Footnote23

CADTH's customers have varying capacity to access or conduct HTA.^Footnotej Provinces with significant HTA capacities see CADTH as a partner, while in the case of provinces with low or no HTA capacity, CADTH has acted as the de facto local HTA organization. For example, New Brunswick's Drugs and Therapeutics Committee turned to CADTH when it needed relevant, specialized evidence to inform and support its decisions. In addition to implementing initiatives to learn about CADTH's drug review processes, the Committee based its review decisions on CADTH's independently reviewed evidence whenever possible. Similarly, the BC Health Technology Review, which focuses on medical device decision-making, has specifically required the inclusion of a CADTH report to support each business case under consideration. As previously stated, CADTH's HTAs help P/Ts who do not have the capacity to carry out their own reviews. Prior to 2003, provinces did have review boards and the process was financially draining and inefficient in that reviews were repeated by multiple provinces.^Footnote24 Similarly, the CDR represents an effort to reduce duplication and maximize consistency and quality of assessments being used to help decision making across the country.^Footnote25 Public drug plans benefit directly from the CDR, recommendations and advice, as a common review reduces duplication across drug plans, while facilitating consistent and timely decision-making across the country.^Footnote26,^Footnote27

CADTH's role as both a producer of evidence and a knowledge mobilizer in the specific area of health technologies (both drug and non drug) is complementary to the role of the Canadian Institutes of Health Research's DSEN. CADTH produces evidence and performs health technology assessments using the current available research. It also acts as a knowledge mobilizer to translate and exchange that evidence to decision makers to inform policy and practice. DSEN supports research and researcher capacity^Footnotek in the area of drug safety and effectiveness, it does not carry out research on non-drug technologies, nor does it assess the cost-effectiveness of drugs. DSEN research is often used by CADTH to inform their HTA processes. Of note, the 2015 Advisory Panel on Health care Innovation recommended that CADTH assume the responsibilities of DSEN which supports post-market research on the safety and cost-effectiveness of drugs, "given the natural affinity of this work with CADTH's mandate".^Footnote28

4.4 Performance: Issue #4 - Achievement of Expected Outcomes (Effectiveness)

In this section, we outline the extent to which key program outcomes have been achieved. Given the difficulty in assessing long-term outcomes, our focus for this evaluation has been on the key immediate and intermediate outcomes that will lead to these long-term outcomes. Therefore, with respect to the long-term goal of having HTAs inform every health technology decision, we have examined CADTH's performance in increasing awareness and understanding of HTAs and optimal use of evidence in addition to evidence for listing decisions; increasing transparency across jurisdictions; increasing use of evidence-based information; improving coordination of HTAs; and improving collaboration among health care system stakeholders.

4.4.1 To what extent have the immediate and intermediate outcomes been achieved?

Increased awareness and understanding of HTA and OU evidence

CADTH has made positive contributions to building receptivity and awareness for health evidence; however, awareness of some new and traditional products and services was not consistently high across stakeholder groups.

CADTH has adopted new customer and knowledge mobilization approaches, accompanied by efforts to strengthen some key products and services (i.e., reviews of non-drug technologies). For example, CADTH's Customer Service Strategy presents 14 concrete actions aimed at providing clarity and strengthening staff customer-service skills. These actions were designed to enable CADTH to assess and adapt products and services in response to evolving customer needs.^Footnote29 In 2014, the creation of the Knowledge Mobilization and Liaison Officer (KMLO) team reorganized and consolidated two previously separate functions, with increased emphasis on addressing issues around awareness-building, knowledge mobilization, capacity building and support initiatives, outreach and impact tracking, and customer follow-up and value for money.

Interviewees noted that CADTH implemented changes in the products already offered, as well as the topics covered, to maintain or improve efficiency. For example, in response to the need for drug and companion device assessments as well as the integration of ethical, economic and implementation considerations, the more operational topics were assigned to the Formulary Working Group and the assessment of classes of drugs were assigned to the Optimal Use Working Group, thereby making the best use of the available expertise. In response to emerging demands for medical device assessments, CADTH started the Health Technology Expert Review Panel (HTERP) in 2011, giving it a mandate to provide recommendations.

Despite this activity, it was clear through the interview process that external stakeholders were insufficiently aware of certain products and services, and that these products and services were not always well understood. Suggestions were made to map CADTH's products and services regarding both drug and non-drug technologies. Interviewees and case study participants suggested the need for CADTH to better communicate its new orientations and priorities both internally and externally.

Some interviewees also suggested CADTH should continuously monitor factors that affect the visibility of its work around drug and non-drug technologies. For example, health care practitioners and decision makers noted the high rotation of staff in health care, and that those who interact with CADTH change quickly; as such, health system managers new to their positions may not be aware of CADTH and the services it offers.

The role and potential value of the KMLO function is well understood among external stakeholders, who expressed a high degree of satisfaction with CADTH's capacity-building and knowledge mobilization activities, although the value of the knowledge mobilization function and the liaison officers activities as a link between CADTH and its customer base were not always well understood internally.

The evidence demonstrated there is an opportunity to better leverage the role of CADTH's liaison officers. According to external stakeholders, liaison officers facilitate the identification of customer needs and help users to access CADTH's products and services. The view was that these qualities help in enhancing clinician engagement processes. Liaison officers have also made it possible to rationalize resources at CADTH, as they frequently redirect customers to existing knowledge products-available through CADTH's website, for example-so that it is possible to meet new demand by leveraging previous work. Liaison officers also fulfil an important intelligence gathering function and are able to provide insights into local political environments.

According to internal key informants, by contrast, the fact that the knowledge mobilization function and the liaison officers work as a link between CADTH and its customer base appears to not be very well known or well integrated with other activities within CADTH. To some extent these functions were perceived as an unnecessary intermediary between customers and the units responsible for providing certain products or services.

Increased transparency across jurisdictions

CADTH exhibits a high degree of transparency and accessibility of its knowledge products. While this has enhanced its credibility, some groups identified a need for additional transparency around the therapeutic and device review discussions.

According to the literature review and interviews, a national-level HTA process plays a significant role in promoting consistency and transparency in health care funding decisions; it provides some minimum uniformity to analytical methods and inputs. Stakeholders interviewed recognized CADTH's high level of transparency and positioning as a pan-Canadian HTA organization. The accessibility (via the website) of the inputs and outputs of reviews, including final reports, was greatly appreciated and contributed to CADTH's credibility. Despite this high degree of credibility, some stakeholder groups wished for more transparency on the context and content of the discussions that take place during drug and device review processes.

Increased utilization of evidence-based information in health care decision making and increased awareness of evidence for listing decisions^Footnotel

There was strong evidence that CADTH's products and services are being used. CADTH has created awareness, understanding and receptivity for health evidence that has informed policy decisions and clinical practice across jurisdictions and individual health practitioners.

The value of CADTH's products and services can be determined in large part by the extent to which they are used by customers. Multiple lines of evidence showed a high level of usage of CADTH's products and services in informing clinical practice, coverage decisions and other policy-making decisions. For products with recommendations, more specifically formulary reviews, administrative data indicated a 90% congruence level between recommendations included in CDR reports and actual uptake by participating drug plans over a 5-year period. The variability observed in uptake levels across jurisdictions can be explained by the timing of formulary decisions made in different jurisdictions.

CADTH's products are serving their intended purpose. In the case of the Rapid Response Service (RRS), customers reported that the purpose of their most recent request was to inform clinical practice decision-making (45%) or policy decision-making (40%). Some variation in the specific use of RRS was evident across jurisdictions. Survey respondents in Alberta, British Columbia, Saskatchewan, and Newfoundland and Labrador used RRS reports to inform policy decision-making as a primary purpose, while survey respondents in the federal government, British Columbia, Alberta, New Brunswick and Saskatchewan used RRS reports to inform clinical practice decision making as a primary purpose.

In regards to capacity building activities, a high degree of satisfaction with the content and format was expressed by those attending CADTH's workshops. Due to differences in the audiences and nature of the capacity building activities conducted by CADTH, it is less straightforward to draw definite conclusions on the actual capacity of participants to mobilize their newly acquired knowledge.

Notwithstanding stakeholders' high level of satisfaction with CADTH's current offerings, the interview evidence suggested that there is room to improve the positioning of some products and services among the customer base. For instance, interviewees commented that the Horizon Scanning^Footnotem function has been useful mainly for those jurisdictions that are more advanced in terms of HTA capacities.

Users of CADTH's products and services were highly satisfied with their quality, utility, relevance and credibility. There were differences in the way customers used CADTH's products depending on their capacities to conduct HTA or to implement recommendations.

Users of CADTH's products and services were highly satisfied with the quality, utility, relevance and credibility of CADTH's current offerings. For example, both survey participants and interviewees recognized the RRS as a service addressing a large range of needs in the decision-making process. In the case of survey participants, 99% of them stated they intended to submit new requests to RRS, and to recommend the service to others.

Generally, CADTH has a positive reputation. Customers with limited HTA capacity rely on CADTH to underpin their decision-making process and find its contribution to be very helpful in addressing their needs for a substantive evidence-base. Those customers with substantive in-house HTA capacities have recognized CADTH's input into their decision-making process. Collaboration improved, particularly among those jurisdictions with HTA capacity, with one outcome being less duplication of work. According to interviewees, an increasing number of jurisdictions no longer re-examine the products that come through CADTH on a regular basis for formulary management at the jurisdictional level. Case study participants recommended a regular update of environmental scans so they could be incorporated into any emerging information or policy changes within the country.

CADTH has improved timeliness in responding to customers' requests, but it could have ensured better use of its products if information were available when listing and price negotiations were being made.

The lack of meaningful administrative data on the overall timeliness of CADTH's delivery of products and services meant this evaluation had to rely on alternative lines of evidence in the form of the online survey, case studies and interviews. According to these lines of evidence, over the time frame covered by this evaluation, CADTH's customers generally felt that the response time to their requests was adequate or improving. Customers acknowledged that the organization has made efforts to improve timeliness more recently. CADTH was successful in addressing a backlog in CDR production over 2013-2014, although this required the use of some financial reserves to cover the cost of producing reports over and above planned capacities. CADTH has since implemented application fees to help finance an increase in the number of drugs CADTH reviews annually. These fees supplement existing federal, provincial and territorial funding.

Case study evidence illustrated how timeliness can influence the uptake and impact of CADTH's knowledge products; for example, the timely arrival of a report with recommendations may make a difference during price negotiations between drug plan managers and industry, and could provide stakeholders with the evidence they need to choose between several technologies. According to interviews, CADTH reviewed three related oral anticoagulants over the span of several months: Dabigatran and Rivaroxaban as part of a Therapeutic Review; and, slightly later, Apixaban as a Formulary Review. Listing and price negotiations were made by jurisdictions informed by the Therapeutic Review. However, these decisions and price negotiations may have gone differently had they also been informed by an assessment of Apixaban (i.e., if the Therapeutic Review had been conducted concurrently with the Formulary Review).

Customer interviewees expressed concerns about the timeliness of some therapeutic reviews and OU products as they either aligned somewhat poorly with the time frame decision makers had to address particular issues, or in the case of non-drug assessments, were affected by the fast pace of technological change. Interviewees suggested that one method for improving the balance between these decision-making requirements and the quality of the information needed to produce a timely report would be to offer a service somewhere between RRS and the more detailed reviews (e.g., therapeutic reviews, OU), depending on user needs and whether other relatively similar products have already been assessed.

In Canada, there was no single organization that could be considered a perfect substitute for CADTH. In the absence of CADTH, customers would either develop in-house capacities, or attempt to mobilize other jurisdictions' capacities, resources from universities or online sources.

Canada's HTA market has a multiple niche structure. Within this, CADTH has played a role in strengthening pan-Canadian coordination and harmonization, as well as serving as a link between different users and producers. Multiple lines of evidence showed that no other organization in Canada is able to assume such a role. For instance, survey participants indicated that CADTH offers quality products and services for which there is no immediate substitute; the RRS is one example of this with 87% of survey respondents noting that no other organization provides a similar service as CADTH's RRS. When assessed on a variety of different criteria, CADTH was seen as better or much better by a number of respondents when compared to other organizations. Of the 13% that mentioned that there are other organizations that provide a similar service, CADTH's RRS compares favourably with regards to timeliness, relevance, credibility and comprehensiveness of its reports. In terms of timeliness, 53% of survey respondents noted that the RRS is much better or better than similar services provided by other organizations. As for the relevance, credibility and comprehensiveness, 39%, 32% and 42% respectively found that the RRS is much better or better than similar services offered by other organizations.

CADTH was one of the few HTA organizations (European network for HTA, National Institute for Health and Care Excellence, Scottish Medicines Consortium) that has been proactive in the systematic implementation of patient engagement as part of its HTA processes, making CADTH an international leader on this front. Incorporating the patient perspective lends additional credibility to CADTH's products and services and increases the likelihood of uptake by customers.

The literature review documented the growing interest of policymakers, health care practitioners and patient groups in organizing and empowering patients to facilitate their involvement in and influence on HTA and health decision making more broadly. The intent is to make the assessment of health technologies more open to and inclusive of patient needs through enhanced interaction between HTA producers and patients. Documented and case study evidence asserted that patients seldom perceive the value of a given piece of health technology in isolation; rather, their opinions tend to reflect their experiences with the health care services they receive, and the extent of their knowledge about alternatives for treatment. Both documented and interview evidence noted the challenges of including patient inputs in HTA activities: patients have diverse interests in, and knowledge about health technologies, while there is also influence from industry lobbying and marketing strategies.

According to interviewees and the literature review, CADTH was recognized as one of the few HTA institutions where patient engagement has been implemented in a systematic way, making CADTH an international leader on this front. Patient engagement activities include the following:

• Adopting mechanisms for collecting patient input into CADTH's decision-making process through the CADTH website.
• Including submissions from patient groups as input into the production of drug reviews, and including information on patient perspectives from the literature into the production of HTA and device optimal use products. For example, patients can provide details of their experience with a specific drug.
• Opening up membership of the Canadian Drug Expert Committee to representatives of the general public who have familiarity with the health system.
• Providing a secretariat for the Patient Community Liaison Forum, which allows CADTH to engage directly with the patient group community to share information and gather feedback on patient engagement processes and supports.
• Sharing knowledge and contributing to capacity building for patient engagement in Canada. For example, in collaboration with CADTH, the Canadian Cancer Action Network has developed a tool to include public input into their evaluation processes. This has resulted in the role of the HTA navigator - a patient who assists patient groups in writing up input for submissions to the CADTH pCODR program. While this practice is currently experimental, it could become more permanent.

Interviewees recommended some areas for further improvement in CADTH's efforts towards patient engagement.

• Improve the definition of patient, in order to make a clear distinction between patient and public representatives as different stakeholders.
• Identify the optimal combination of methods by which patient input can be obtained.
• Build on mechanisms for balancing between specific lobbies and patients acting as individuals.

In moving forward, according to both the literature review and interviews, international organizations like Agència de Qualitat i Avaluació Sanitàries de Catalunya (AQuAS) in Spain or National Institute for Health and Care Excellence in the UK offer examples of good practice in the area of patient engagement. The following initiatives undertaken by international HTAs may be of interest to CADTH:

• European network for HTA: Integration with regulatory bodies from the clinical trial registration all the way through to post-marketing. This provides the kind of early stage evidence needed by the HTA body so that when a drug is approved by the regulatory body, clear recommendations and post-market monitoring processes (if needed) are available.
• National Institute for Health and Care Excellence: Patient engagement and patient trainees as well as dual- or multi-technology reviews.

Improved coordination of drug and other health technology reviews

In line with F/P/T governments' commitments to HTA collaboration and their goal of improving the efficiency and cost-effectiveness of public health care service provision, CADTH has introduced initiatives to improve coordination and knowledge sharing between HTA producers by adopting a new strategic approach to partnerships, and has changed its traditional relationships with some customers.

The federal government has maintained a commitment to address priority health issues by supporting key pan-Canadian organizations.^Footnote30 The government's stated intent remains to "engage with stakeholders from the provinces and territories to share knowledge and information related to health technology management, and to minimize duplication of efforts with respect to the introduction, diffusion and utilization of health technologies."^Footnote31

In this context, evidence on CADTH's ongoing organizational transformation showed efforts to adapt and respond to emerging trends in its operational environment, while keeping with the goals of rationalization of public expenditure. For example, CADTH improved coordination and knowledge sharing by adopting a new strategic approach to partnerships to help broaden the reach of HTA in Canada. This included support to the Health Technology Policy Forum (HTPF) to produce an environmental scan of existing personalized medicine policies and practices across Canada, development of a common typology of terms used in the personalized medicine arena, and development of a common assessment framework for companion diagnostics.

Improved collaboration among health care system stakeholders, including other producers and users of evidence

By fostering collaboration among health stakeholders, CADTH has contributed to reducing duplication in HTA processes in Canada. CADTH has improved collaboration with most of the identified stakeholders, particularly those that have long-standing HTA capacities (e.g., Health Quality Ontario and Institut national d'excellence en santé et en services sociaux).

This evaluation found documented evidence on CADTH's efforts to respond to its changing operational environment, including through participation in debates around novel methodologies and approaches to HTA, and collaboration with peer organizations in Canada and internationally. According to CADTH strategic documentation, collaboration opportunities have been identified in areas related to common methods for producing HTA in Canada; approaches to enable HTA information sharing (e.g., a searchable repository for HTA reports produced in Canada); Canadian HTA bodies and Health Canada working together on approaches to the regulation and assessment of health technologies; use of pre- and post-market HTA evidence; identification of key entry and influencing points for health technologies to the health care system; and use of regional and hospital-based HTA to inform local decision-making.

An example of progress around Canadian partnerships was the launch of the pan-Canadian Health Technology Assessment Collaborative (pCHTC), formed in 2012 with the goals of sharing best practices, minimizing duplication of effort through the sharing of information, and identifying and contributing to joint initiatives. The pCHTC brought together policy decision makers (regional health authority executives, provincial health departments, and Health Canada), the academic community (University of Alberta), HTA producers (Health Quality Ontario, Institut national d'excellence en santé et en services sociaux, CADTH, Institute of Health Economics), and other stakeholders (Policy Forum and Health Technology Analysis Exchange). According to interviews, Health Quality Ontario and Institut national d'excellence en santé et en services sociaux have shown a willingness to tighten collaboration with CADTH; beginning in 2015, the three organizations initiated efforts to align agendas and methodologies and to avoid assessment duplications.

CADTH's collaboration with provincial HTA organizations has facilitated consistency in the work across jurisdictions. The most evident outcome of this was the decreasing number of jurisdictions with a need to re-review drugs before their inclusion in the provincial formulary.

Collaboration with provincial HTA organizations was identified as a mechanism to enhance consistency and to some extent reduce duplication of HTA work across Canada. Interview evidence indicated progress has been achieved with regard to drug reviews, where CADTH has played an important role in strengthening pan-Canadian coordination. The main perceptible outcome was the decreasing number of provincial and territorial governments needing to re-review drugs before their inclusion in formularies. Interviewees mentioned some positive dynamics around collaboration between major HTA organizations for the production of non-drug technology assessments; in some cases, this has led to a division of labour at the operational level. For example, the Institute of Health Economics did an economic assessment for a technology, and CADTH did its clinical assessment.

Interview data presented opportunities for CADTH to further engage with F/P/T customers and to support them in implementing the optimal use of health technologies in a broad sense. For instance, CADTH could explore ways to provide practical support in the implementation of recommendations, aligned with the conditions and capacities of jurisdictions. It was hoped that CADTH could work with provincial HTA organizations so that in addition to a sound scientific evidence-base, reviews can include contextualized information about budget impact, local patient perspectives and local policy priorities. According to interviewees, contextualization could be enhanced by bringing in more expertise from the field on, for example, clinical practice on the sequencing of various therapies for a particular treatment of disease, or for specific environments (e.g., rural or remote areas) or patients. Users would also benefit from knowledge about possible constraints associated with the health care system.

CADTH has led international debates and initiatives around novel methods for HTA, and has collaborated with international HTA organizations and networks (e.g., European network for HTA, International Network of Agencies for HTA).

Interviewees and documented evidence corroborated CADTH's efforts to maintain collaboration with international HTA organizations and networks, including the European network for HTA and International Network of Agencies for HTA. An active participation in international fora has increased the organization's visibility and leadership in an otherwise largely European-dominated space. Examples of outputs from international collaboration include setting up communities of practice to discuss the proliferation of expensive drugs, an ethics group, a patient involvement group, an internship program, methodological work, harmonization and standardization of HTA terminology, and mechanisms to facilitate information sharing. Additional short-term outcomes include CADTH staff members' participation in international HTA meetings, contribution to international conferences, workshops and peer-reviewed publications, or assuming leadership positions within some of these organizations. Through CADTH's international collaborations, awareness is raised regarding what the organization does and its reputation for producing quality work. This reputation attracted the Australian Institute for Safety, Compensation and Recovery Research to engage CADTH with a paid contract to develop a horizon scanning program tailored to spinal cord injury for the Institute.

CADTH has demonstrated leadership in identifying drug and non-drug topics of importance for customers through broad consultations with stakeholders. The evidence was insufficient to assess the extent of influence on the quality, consistency and utility of CADTH's knowledge products.

Interviews and document data showed that CADTH has followed a proactive approach of engaging in broad consultations with different stakeholders to identify emerging topics of interest.^Footnoten These topics are expected to reflect stakeholders' priorities and needs around HTA, including but not limited to device-related topics. In addition to topic intelligence gained through regular outreach with customers at all levels of the health system, Consultations on topics have taken place through informal (e.g., discussions with health decision makers by CADTH liaison officers) and formal mechanisms. Some examples of the latter are as follows:

• Open calls for input and feedback on CADTH's website. With medical devices, the list of topics is examined periodically according to a scoring tool, which has been available on the website since November 2015.^Footnote32 Topics are then ranked and validated (with respect to capacity and priority) by CADTH senior managers, liaison officers and the HTERP. Customer interest and commitment are key criteria.
• Consultations with HTA producers and other key stakeholders leading to the development of a pan-Canadian HTA collaborative model for the production, dissemination and uptake of HTA information.
• Focus group sessions held with customers from across the country, including senior-level representatives of the health care systems involved in decision-making related to drugs and devices. Information collected about customer experiences and perspectives on key priorities has informed CADTH's topic identification processes.
• Horizon scanning.

Interviewees welcomed the strategy of consulting with stakeholders to hear and learn about their needs around HTA. This was expected to strengthen CADTH's influence on policymaking. However, interviewees also expressed their wish for a more rational use of resources, by CADTH finding alternative ways to conduct consultations by clearly defining criteria for selecting consultation participants. More importantly, the suggestion was made to strike a balance between CADTH's strategic plans and consultations that take into account CADTH's own strategic priorities.

4.5 Performance: Issue #5 - Demonstration of Economy and Efficiency

The Treasury Board of Canada's Policy on Results (2016) and guidance document titled Assessing Program Resource Utilization When Evaluating Federal Programs (2013), define the demonstration of economy and efficiency as an assessment of resource utilization in relation to the production of outputs and progress toward expected outcomes. This assessment is based on the assumption that departments have standardized performance measurement systems and that financial systems link information about program costs to specific inputs, activities, outputs, and expected results.

Observations on Economy

CADTH's ability to provide economic value to its customers could be enhanced with improvements to the process of strategic direction setting.

CADTH has changed its traditional relationships with some stakeholders. While capacity development workshops aimed at customers remain available free of charge, products and services such as the CADTH symposiums, skills development workshops, and the Scientific Advice Program are provided on a cost-recovery basis. Additionally, in 2014 and 2015, CADTH introduced application fees for CDR and pCODR submissions to help finance an increase in the number of drugs CADTH reviews annually. The fees supplement existing F/P/T funding. This initiative was not free of controversy, though. Some stakeholders initially perceived this would negatively affect CADTH's independence. However, to date, there is no evidence to support these concerns.

CADTH has insufficient human and financial resources to meet the increased demand for broader therapeutic reviews and the growth in both drug and non-drug technologies. The documented evidence and interviews recommended that a long-term resource strategy should help CADTH meet performance requirements while maintaining accountability and transparency. Interviewees in particular suggested the pertinence of making choices about which issues are of importance and to plan resourcing of CADTH's activities accordingly. As previously discussed, CADTH introduced application fees for industry submissions and resubmissions to the CDR program. While the expected share of income from industry fees is expected to increase significantly relative to other funding sources, according to CADTH's internal documentation, this fee approach remains a short- to medium-term solution.

CADTH operated at or near to its capacity for producing products and services over the time frame for this evaluation. Although data on the extent of CADTH's human resources allocations has not been captured for a sufficient amount of time for the evaluation to make a detailed analysis of resource allocations, it is reasonable to expect that a significant increase in the production of some of CADTH's more complex and technically demanding products would present a staffing challenge.^Footnoteo Resources would need to be reallocated from other activities. While improved priority setting may help in dealing with some of these growth pressures, the data suggested there is a need for careful planning of CADTH activities based on the available resources.

TheF/P/T governments primarily fund CADTH. The Government of Canada provides the majority of CADTH's funding and has committed to providing CADTH with up to $80,631,924 in funds to support CADTH's work as outlined in the Contribution Agreement effective April 1, 2013 to March 31, 2018.^Footnote33 As previously mentioned in section 2.4, there was a relatively small reduction in federal government funding of 5% phased in over 2012-2013 and 2013-2014; one-time federal funding for work on isotopes ending in 2011-2012; and the introduction of industry application fees in 2014-2015. The Government of Canada provided $65.4 million to CADTH over the four years covered by this evaluation. Table 3 shows CADTH's revenue from 2012 to 2015. Health Canada funding accounts for between 58% and 81% of the total CADTH revenue each year. According the CADTH financial statements, all P/T governments (except Quebec) provide funding to CADTH with Ontario, British Columbia and Alberta providing the largest contributions.

As previously seen in section 2.4, CADTH had a budget of $66.2 million over the years covered by this evaluation. Contributions from federal and P/T governments amounted to $64.1 million, and other income included symposium and workshop revenues, interest revenues, other service revenues for work done for other organizations and the introduction of industry application fees for CDR in 2014-2015. 

Table 4 highlights the variance between planned and actual expenditures for CADTH. As is demonstrated in table 4, over the last four years CADTH has reported spending between 90% and 107% (with an average of 100%) of its planned allocations. In 2013-2014, CADTH were not able to complete all the work in the OU/HTA bucket and funding was carried forward to work on improving processes. Salaries increased both in 2014-2015 and 2015-2016 for a variety of reasons including: pCODR became part of CADTH; a number of previously open positions were filled; there was a revitalization of the medical devices program; and an increase in formulary review submission rates and the introduction of industry fees.

Observations on Efficiency

Over the timeframe of the evaluation, CADTH made improvements to the efficiency and effectiveness of the governance structure and practices by changing the composition of the Board of Directors, changing the reporting structure and reorganizing the committee structure; however, there are opportunities for further enhancements.

Governance

Improvements made to CADTH's governance structure and practices have helped the organization to remain a relevant HTA provider within the landscape of Canadian governments' priorities around HTA. A case in point was the change made to the composition of the Board of Directors in 2011, which resulted in more diverse representation and helped to raise CADTH's reputation and trust among stakeholders. This ongoing transformation also resulted in a change in governance, in which expert and advisory committees report to the Chief Executive Officer, rather than directly to the Board. The evidence gathered through this evaluation suggests that a review of the CADTH committee structure (including those groups for which CADTH provides Secretariat support) and connectivity of the committees with the Board may reveal opportunities to optimize inputs to strategic direction setting.

CADTH is governed by a Board of Directors elected by the Members of the Corporation who are the Deputy Ministers of Health of participating federal, provincial, and territorial governments (as CADTH's corporate members). The 13-member Board is composed of an independent chair, seven jurisdictional representatives, and five non-government representatives from health authorities, academia and the general public. While Quebec does not provide funding to CADTH, a representative is appointed as an observer to the Board of Directors.

There are many panels, committees and working groups that play a role in the work of CADTH. They consist of experts in various health fields, and essentially facilitate the production of CADTH outputs, which are disseminated to customers and other users. These various groups^Footnotep are identified briefly below^Footnote34 (see Appendix 4 for more information).

1. Drug Policy Advisory Committee;
2. Drug Policy Advisory Committee (DPAC) Formulary Working Group;
3. DPAC Optimal Use Working Group;
4. Canadian Drug Expert Committee;
5. Provincial Advisory Group;
6. pCODR Advisory Committee;
7. pCODR Expert Review Committee;
8. Health Technology Expert Review Panel;
9. Pharmaceutical Directors Forum;
10. Policy Forum;
11. The Health Technology Analysis Exchange;
12. Pan-Canadian Health Technology Assessment Collaborative.

The various lines of reporting and interactions are illustrated in Figure 1.

Figure 1: CADTH governance structure^Footnoteq

CADTH has improved its governance, largely due to a reorganization of staff and committees in 2009, but some streamlining and enhanced communication would be helpful in making specific activities more efficient. The evaluation identified some areas of duplication between certain committees. For instance, between the DPAC, the DPAC Optimal Use Working Group and the DPAC Formulary Working Group. With a certain amount of cross-membership between the groups, it may be possible to uncover efficiencies by streamlining communications or certain functions among the groups.

Compounding this issue is the fact that in addition to these three expert committees and two advisory committees (with three subgroups) there are an additional 16 working groups, panels and other fora (many of which have been mentioned above) where CADTH has oversight or fills a secretarial role. Analysis of the membership of these 20 expert and advisory bodies shows there are 22 individuals who sit on more than one committee, including several who sit on up to three. The advisory bodies meet at varying intervals-some monthly, some quarterly, some annually-with some meetings by teleconference and others held in person.

Interviewees mentioned that a recent alignment (April 2016) of CADTH's recommendation framework will likely contribute to better alignment in the levels of effort. Nonetheless, the number and variety of drugs to be reviewed has to be taken into consideration in the allocation of the resources needed to perform the reviews (the workloads of committee members could also be taken into account).

Interviewees noted challenges experienced by some jurisdictional representatives sitting on more than one drug committee. Time commitments and the management of travel-related expenses were reported as a burden, particularly among representatives of jurisdictions with low HTA capacities, where there may also be only few individuals with the appropriate expertise to represent those jurisdictions.

Data on CADTH's planned and actual production showed that it operated at a level close to full capacity.

Data on CADTH's planned and actual production by major product/service line for the period 2012-2016 suggested that the organization operated at a level close to full capacity (Table 5). Actual production differed across products and service lines, while some variation was evident between planned and actual output, in part reflecting fluctuations in market demand. Optimal use drug therapeutic class reviews are a case in point, with production between 2012-2013 and 2015-2016 consistently below capacity. In contrast, the RRS' production was well above the planned production for 2014-2015 and 2015-2016. Further, the volume of environmental scans was dependent on the level of customer demand; actual output was 20%-50% above plan in 2015-2016. Despite these variations, CADTH had enough flexibility to adjust production levels according to market fluctuations.

In terms of customer base, the federal government, as a single entity, is the largest consumer of CADTH's products and services ^Footnoter by volume, using 250 of CADTH's outputs from 2012-2016. Saskatchewan, British Columbia, Alberta and Newfoundland and Labrador round out the top five requestors of products and services with 213, 189, 142 and 71 respectively. Table 6.

A comparison between the number of CDR submissions received and reviewed (including Canadian Drug Expert Committee (CDEC) recommendations) completed from January 1, 2012 through March 21, 2016 (Figure 2) revealed a mismatch between production capacity and demand for CDRs. This mismatch may be due to the fact that submissions can be received in one calendar year and reviewed in the next. It can also be because submissions can be withdrawn, suspended or rejected.

It is still too early for there to be any significant evidence on the effect that collecting fees might have on the organization’s financial stability; however, this initiative highlighted the need for a mechanism to manage submission volume so as to minimize the possibility of future queueing, as well as to mitigate the impact on stakeholders by ensuring that those drugs with an expected greater potential to improve the health system would not be delayed.

The data also signal the challenging position in which CADTH has found itself in terms of responding to emerging issues in its operating environment. As innovation in health technologies increases, so does the demand for certain products and services, leading to increased pressure to optimize resource allocation across major product and service lines.

Figure 2: Canadian Agency for Drugs and Technologies in Health Common Drug Review submission volumes and review activity by calendar year (January 1, 2012 to March 21, 2016)

Observations on the Adequacy and Use of Performance Measurement Data

Data collection efforts at CADTH are most successful at the activity and output levels. Systems are in place to track the extent of CADTH's contribution to long-term outcomes (e.g., improved health outcomes, policy changes, policy coordination); however, this data is not readily available from the health system. These could not be assessed due to an absence of any secondary data and lack of informed stakeholder opinion.

CADTH's ongoing organizational transformation included a number of key strategic elements, each designed to support increased profile and impact for HTA in Canada. For example, the decision to streamline the KMLO functions was intended to integrate and leverage CADTH's investments in outreach actions and functions, including stakeholder engagement, partnerships, conferences, government relations, knowledge exchange, marketing, and web/new media development. The streamlining of the KMLO function included a revamping of data collection processes and a steady move towards the present KMLO impact database.

However, this evaluation found there was no centralized repository or system for storing, retrieving, processing and quality assurance of performance data at CADTH. In its current form, data collection and performance tracking seem to reflect a disconnected view of the organization; while individual units collect data and report on performance, there is no consistent way to make the links between the data and performance indicators as per CADTH's Performance Measurement Strategy.

This evaluation tried to obtain data on long-term outcomes such as social, economic and health/health system benefits, as well as examples of changes in policy practices or enhanced policy coordination. Based on the information collected and interaction with different stakeholders during the data collection process, it was determined that such data do not exist in any form that would enable them to be linked back to CADTH. Overall, for the time frame covered in this evaluation, it was too early to observe any major health care changes associated with recent activities undertaken by CADTH.

The individuals interviewed as part of the interview and case study lines of evidence were unable to qualify or quantify the long-term impact of having taken up or used CADTH's products, despite the fact that they generally found value in those products. Nevertheless, as previously indicated, interview data revealed that enhanced coordination between CADTH and provincial HTA organizations may have helped reduce duplication in HTA work. However, because the number of such reductions could not be precisely quantified, it was not possible to establish a rough value of the actual benefits that could be associated with this observed outcome. The evaluation arrived at a similar conclusion in the case of outcomes related to CADTH's contribution to the harmonization of international HTA activities-for example, it was mentioned that Scotland no longer repeats reviews of devices that CADTH has already undertaken.

Based on interviews, case study data and the review of administrative data, the evaluation found that in order to assess the economic value of CADTH products, additional effort is needed to determine both the appropriate outcomes to be measured (e.g., the amount of money saved by adopting a more cost-effective technology, or by reducing the overuse of a product) and how to capture CADTH's contribution relative to other factors that may have had an influence on the specific outcome. The evidence collected also identified some limiting factors for the uptake and impact of CADTH's products and services, including the timeliness of the reports (i.e., therapeutic review/OU HTA projects), political buy-in and support, the nature and pace of regulatory changes and the incidence of a condition in a given population. These should be taken into account when assessing CADTH's contribution to long-term outcomes.

5.0 Conclusions

5.1 Relevance Conclusions

5.1.1 Continued Need

There is a continued need for CADTH’s assessments of drugs and health technologies as these are a source of information in evidence-based decision making. CADTH activities are important as there is an increased demand for health technology assessments resulting from innovation in health technologies and convergence of drug and non-drug technologies.

5.1.2 Alignment with Government Priorities

As outlined in a variety of strategic documents, protecting the health and safety of Canadians remains a priority for the Government of Canada. The Government funds pan-Canadian organizations such as CADTH to support health system innovation and sustainability which also remain priorities for the Government as evidenced by their inclusion in recent federal agenda-setting documents such as the Speeches from the Throne, Budgets and the 2015 Minister of Health mandate letter.

5.1.3 Alignment with Federal Roles and Responsibilities

Health Canada has a clear federal role to act as a catalyst for innovation, influencing health care policy and providing evidence-based information to health care decision makers. The activities carried out by CADTH are aligned with the Minister’s legislative authorities under the Department of Health Act (1996).

Health Canada makes investments to help the P/Ts accelerate change in areas of shared priority. Health Canada supports CADTH’s mandate to deliver reliable, timely, evidence-based information to Canada’s health care leaders on the effectiveness and efficiency of health technologies. This information is particularly useful to P/Ts which do not have the capacity to carry out their own reviews.

CADTH’s role as both a producer of evidence and a knowledge mobilizer in health technologies (both drug and non drug) is complementary to the role of the Canadian Institutes of Health Research’s DSEN. As CADTH produces evidence and performs health technology assessments using the current available research. It also acts as a knowledge mobilizer to translate and exchange that evidence with decision makers to inform policy and practice. Meanwhile, DSEN supports research and researcher capacity in the area of drug safety and effectiveness, it does not carry out research on non-drug technologies, nor does it assess the cost-effectiveness of drugs. In addition, DSEN research is often used by CADTH to inform HTA processes.

5.2 Performance Conclusions

5.2.1 Achievement of Expected Outcomes (Effectiveness)

While CADTH has made positive contributions to building awareness and understanding of health technology assessment and optimal use evidence, awareness of some of its new and traditional products and services was not consistently high across stakeholder groups. For example, the role and potential value of knowledge mobilization activities are well understood among external stakeholders, although the value of the knowledge mobilization function and the liaison officers’ activities as a link between CADTH and its customer base were not always well understood internally. While CADTH has knowledge products that are readily accessible which has helped enhance its credibility, some industry and patient groups have identified a need for additional transparency around the therapeutic and device review discussions.

CADTH helped increase awareness and understanding of its HTA and OU evidence and there was strong evidence that CADTH’s products and services are being used by stakeholders. Users of CADTH’s products and services were highly satisfied with their quality, utility, relevance and credibility. There were differences in the way customers used CADTH’s products depending on their capacities to conduct HTA or to implement recommendations, jurisdictions with limited capacity to carry out assessments have relied more heavily on CADTH’s advice and recommendations to make decisions on coverage. CADTH has improved timeliness in responding to customers’ requests, but it could have ensured better use of its products if information were available when listing and price negotiations were being made.

In addition to its stated outcomes, evidence shows that CADTH is one of the few HTA organizations (European Network for Health Technology Assessment, National Institute for Health and Care Excellence, Scottish Medicines Consortium) that has been proactive in the systematic implementation of patient engagement as part of its HTA processes. Incorporating the patient perspective lends additional credibility to CADTH’s products and services and increases the likelihood of uptake by customers.

In line with F/P/T governments’ commitments to HTA collaboration and their goal of improving the efficiency and cost-effectiveness of public health care service provision, CADTH has introduced initiatives to improve coordination and knowledge sharing between HTA producers, and has changed its traditional relationships with some customers. Further, by fostering collaboration among health stakeholders, CADTH has contributed to reducing duplication in HTA processes in Canada. CADTH has also improved collaboration with most of the stakeholders, particularly those that possessed long-standing HTA capacities (e.g., Health Quality Ontario and Institut national d’excellence en santé et en services sociaux).

5.2.2 Demonstration of Economy and Efficiency

CADTH is primarily funded by the federal, provincial and territorial governments. The Government of Canada committed to providing CADTH with up to $80.6 million from 2013 to 2018 (of which $65.4 million was provided over the four years covered by this evaluation). Federal government funding to CADTH varied from 58% to 81% of CADTH’s total annual funding from April 2012 to March 2016. In addition to federal funding, CADTH receives P/T funding as well as other funding that includes revenue from symposiums and workshops, interest revenue, other service revenue for work done for other organizations and industry application fees for the CDR program.

Over the timeframe of the evaluation, CADTH continued to make improvements to the efficiency and effectiveness of the governance structure and practices (e.g., changes made to the composition of the Board of Directors and reporting changes); however, there are opportunities for further enhancements.

With respect to performance measurement, data collection efforts at CADTH are most successful at the activity and output levels, and systems are in place to track the extent of CADTH’s contribution to long-term outcomes (e.g., improved health outcomes, policy changes and policy coordination); however, this data is not readily available from the health system. These outcomes could not be assessed due to an absence of any secondary data and lack of informed stakeholder opinion.

6.0 Recommendations

The recipient-led evaluation resulted in three recommendations directed at CADTH which can be found on the CADTH website, whereas this departmental-led evaluation did not result in any recommendations directed at the department.

Appendix 1 – CADTH’s portfolio of products and services

Appendix 2 - Logic Model

Figure 3: Logic Model

Appendix 3 - Evaluation Description

Evaluation Scope

The scope of the evaluation covered the period from April 1, 2012 to March 31, 2016, and included various CADTH activities such as HTA, OU and CDR. The evaluation did not cover the pCODR program itself as it was transferred from the P/Ts to CADTH during the evaluation (April 1, 2014).

The evaluation issues were aligned with the Treasury Board of Canada's Policy on Results (2016) and considered the five core issues under the two themes of relevance and performance.

A theory-based evaluation approach was used for this evaluation to assess the progress made towards the achievement of the expected outcomes, whether there were any unintended consequences and what lessons were learned.

The Public Health Agency of Canada and Health Canada Office of Audit and Evaluation acknowledges and thanks all individuals who gave their time and input for this evaluation. In particular, the Office of Audit and Evaluation acknowledges the consulting firm of Science-Metrix which conducted the evaluation of CADTH's activities.

Data Collection and Analysis Methods

Evaluators collected and analyzed data from multiple sources. Data collection started in January 2016, prior to the conclusion of fiscal year 2015-2016.

Literature and document review

The literature review was conducted to position CADTH within a broad reference with respect to Government of Canada and provincial/territorial priorities in health; the health science research context; and the context around the specific field of HTA.

This contextualization was useful to examine issues of relevance primarily, as well as to identify enabling factors and barriers to performance. The bulk of the documentation included in the review was received from CADTH at different stages, including during the planning phase of this evaluation. CADTH provided an annotated document outlining the key considerations that the evaluation was expected to address as part of the analysis of the context of HTA in Canada and internationally. Finally, the evaluation conducted searches for scholarly and grey literature. While this search was bounded by the time frame for this evaluation, the evaluation also found it useful to look at a selection of older documents when these enabled a better understanding of events covered by more recent literature.

An internal and external document review was also conducted to position CADTH within a broad reference of operation, taking into account the Canadian and international context. The evaluation examined documents on CADTH's mandate, strategies, product line operations, products and services, as well as reported results. The review included strategic plans, annual business plans, performance reports, the current Contribution Agreement between CADTH and Health Canada, and other documentation as provided by the project authority. The more purposive document review incorporated internal studies, and external documents related to governance structures and processes. The re-examination of the reports and recommendations stemming from previous CADTH evaluations and associated management responses was appropriate in order to establish a baseline for the assessment of performance over the period 2012-2015. Additional reviews were conducted during the data triangulation phase in order to substantiate some emerging findings related to performance, efficiency and economy.

The literature and documents serving as secondary data sources were inventoried and subject to content analysis using the qualitative data analysis software Atlas.ti (www.atlasti.com). All secondary data sources, as well as primary data products, were uploaded and coded to conduct in-depth analysis by evaluation question and indicator in the DCM. Relevant text from the captured stock of documents was coded using both the deductive and inductive approaches. First, a closed coding structure was developed based on the evaluation matrix. Open coding was also used when other unforeseen topics of interest were identified. The evidence base extracted from the literature review was analyzed, with data summaries and observations prepared as input for triangulation and face validation against other lines of evidence.

Administrative and financial data analysis

The data review conducted as part of this evaluation included the datasets listed in table 7, as provided by CADTH's project authority, with information covering the period 1 April 2012 to 31 December 2015. The data collection and analysis portion of the evaluation began in January 2016, prior to the conclusion of the 2015-2016 fiscal year.

The financial, administrative and performance database review included an analysis of quantitative data collected by CADTH over the evaluation time frame. The scope of the administrative database review was to gain insight into the administrative processes leading to the identification and production of CADTH products and services with emphasis on production capacity, timelines, turnaround times, dissemination media and, as much as possible, the uptake of CADTH's knowledge products by intended users. Particular attention was given to the uptake of product lines, which included recommendations - in CDR, for example. The databases were also mined for data regarding outreach and partnership development efforts to enhance knowledge sharing, coordination and collaboration. An important component of the analysis was to focus on the indicators identified in CADTH's Impact and Evaluation Framework, as generated and compiled by CADTH.

The analysis of financial data was intended to gain insights into economy and efficiency issues by examining operations and maintenance costs, overhead costs and resource allocation across the product line portfolios. However, changes made to CADTH's financial reporting practices over the period covered by this evaluation have affected the ability to analyze data over a sufficiently long time series, as needed for a robust assessment of operational efficiency between 2012 and 2016. The dataset used in the analysis was constructed by CADTH based on non-audited financial data and was intended solely for the purpose of this evaluation. A re-constructed financial dataset was required to allow for year-to-year comparisons following a change in accounting practices midway through the time frame of this evaluation. The data showed a breakdown of expenditure by main program area (i.e., HTA, CDR, OU), as well as administration/corporate services. The data are to be used with caution when informing findings on CADTH's efficiency and economy.

The Knowledge Mobilization and the Liaison Officer functions at CADTH underwent a process of reorganization and consolidation in early 2014 the two are now amalgamated under the KMLO Program. The consolidation involved improvements and streamlined data collection initiatives resulting in the present KMLO impact database. For the purpose of this evaluation, two individual datasets covering two different time periods (the one with data for the period before April 2015, and the one covering the period April-December 2015) were provided by CADTH. Comparative data was compiled by the KMLO team to share information on impact for projects spanning the two database time frames.

All of the database reviews included an assessment of data accuracy and continuity over time (if there is a time series), gaps vis-à-vis performance measurement strategies, and their utility for developing a performance story based on the contribution analysis approach and assessing achievements against goals, objectives and targets set out in CADTH's management documents, such as strategic plans and annual business plans. The database reviews provided insight into the validity and reliability of the performance measurement data in support of the evaluation function as per the Treasury Board's guidelines.^Footnote35

The analysis of the databases was carried out using standard data mining techniques with an MS Excel workbook. Each database underwent substantial cleaning with the intention of making the data consistent for the analysis. The evaluation made explicit to the project authority any decisions made in regard to the cleaning and processing of the data used in this evaluation. This was carried out in ways consistent with the definitions and structure of the data contained in the distinct databases provided by the project authority.

Bibliometric data

Although this evaluation was not mandated to conduct a bibliometric analysis of CADTH's activities, bibliometric data compiled by CADTH was useful to inform performance indicators during the data collection phase of this evaluation. For example, the data helped to understand CADTH's work in the context of academic debates, debates in the grey literature and discussions on online media. These activities can be interpreted as contributing to building a culture of receptivity of evidence around HTA and OU of health technologies. The analysis of this database was also carried out using standard data mining techniques with an MS Excel workbook.

Key informant interviews

As per table 8, 60 interviews were held with a variety of groups: internal CADTH staff, CADTH board members, committee members, government mid policy, government senior policy, clinicians, drug plan managers, patient group representatives, international and Canadian HTA producers and industry. Interviews with CADTH personnel and/or individuals, who have played a significant role in the design and delivery of the portfolio of products and services, as well as other strategic activities, provided valuable insight on key evaluation questions related to relevance (e.g., continued need, alignment with governmental priorities), efficiency and lessons learned.

External stakeholders knowledgeable about CADTH's operations and its operational context-such as provincial and territorial health care customers-provided evidence related to the achievement of outcomes, helped gauge the continued need for CADTH's products and services, and identified external factors that influenced CADTH's performance. These interviewees were also asked to identify any unintended outcomes, either positive or negative, that could be attributed to CADTH's products and services. Table 8 presents the distribution of interviewees by group.

In-person and telephone interviews were conducted using an interview guide tailored for each group and that allowed for deviations, prompts and follow-ups when appropriate. All interview transcripts were systematically coded and analyzed by indicator using Atlas.ti. Summaries of key findings by evaluation question and indicator across interviewee groups were then prepared as input into the data triangulation process.

E-survey

Science-Metrix designed, programmed and administered an electronic survey of customers of the RRS. The RRS serves a large base of customers with a diversity of customized sub-products. These sub-products vary in levels of complexity, depth of analysis and time requirements. The survey explored customers' levels of satisfaction and the perceived value they derive from the RRS. The survey questions addressed topics such as the timeliness of the services provided, the completeness of the responses, the credibility of the information, the overall satisfaction with the service provided by CADTH and the utility of the information disseminated. The survey also contained additional questions on the influence of this information on health policy and clinical practice decision-making.

The initial list of potential e-survey respondents comprised 441 individuals who had made at least one request through the RRS since 1 April 2012. The participant names and contact information were provided by CADTH. CADTH liaison officers gave advance notice to the identified survey participants that they would be contacted to respond to the survey. Prior to sending survey invitations, CADTH determined that two (2) of these individuals would instead be interviewed and would thus not be invited to participate in this survey. Additionally, through a vetting question included in the survey, six (6) individuals noted they had not made at least one request to the RRS since 1 April 2012, thus making them ineligible to complete the survey. The final e-survey population was determined to be N=433 individuals.

A total of 146 respondents completed the survey over a period of four weeks, a response rate of 33.7%. The survey findings are generalizable to the entire population with 95% confidence and a margin of error of ±6.6%, though findings cannot be generalized within each jurisdiction.

The e-survey was administered through the Fluid Surveys platform. Respondents accessed it through a unique link, which allowed Science-Metrix to track those respondents who accessed and completed the e-survey in real time to ensure reminders could be targeted. Three email reminders were sent to respondents who had not completed and submitted the e-survey.

Quantitative and qualitative data analyses were undertaken using a combination of tools, which included Fluid Surveys' reporting functions as well as Microsoft Excel. Responses were analyzed by evaluation question.

Case studies

The case studies were selected both to illustrate a variety of outcomes and to cover a broad range of products and activities that are used to inform evidence-based policy and clinical practice decision-making. CADTH identified six topics and corresponding CADTH products/activities for study:

• Urine Matters/Cough Matters - Lab optimization (rapid response service and knowledge mobilization and implementation support)
• New oral anticoagulants (NOACS) for atrial fibrillation - Common Drug Review and Therapeutic Review (optimal use)
• Linagliptin/metformin (Jentadueto) for type 2 diabetes mellitus - Diabetes evidence bundles (knowledge mobilization and implementation support)
• Novel drug therapies for relapsing-remitting MS (RRMS) - Patient engagement (as part of optimal use project)
• Point-of-care international normalized ratio testing (POC INR) - Outreach (optimal use and knowledge mobilization and implementation support)
• Reprocessing of single-use medical devices (SUMDs) - Environmental scanning

Each case study involved a review of literature and documents relative to the specific case, as well as a set of interviews with internal and external stakeholders. Internal interviewees were representatives of CADTH, including KMLO team members, who were involved in the delivery of CADTH products and activities for each specific case. External interviewees were CADTH customers who used CADTH's products to inform health policy or practice changes, including drug plan managers and other provincial or regional health authority representatives. Key findings and conclusions (limited to each case) stemming from the document review and interviews were identified for each case study and included in the evidence base for this evaluation.

Appendix 4 - Advisory bodies that play a role in the work of CADTH

Drug Policy Advisory Committee: The DPAC comprises representatives from the federal, provincial, and territorial publicly funded drug plans, and other related health organizations. DPAC provides strategic advice on drug policy issues and drug topics to CADTH and its Board. Committee members also facilitate effective jurisdictional sharing of drug policy information.

DPAC Formulary Working Group: The DPAC Formulary Working Group includes representatives from the federal, provincial, and territorial publicly funded drug plans and other related health organizations. The DPAC Formulary Working Group provides advice to CADTH on issues related to the CDR process. Working group members also facilitate effective jurisdictional sharing of pharmaceutical information.

DPAC Optimal Use Working Group: The DPAC Optimal Use Working Group includes representatives from the federal, provincial, and territorial health ministries, and related health organizations. The DPAC Optimal Use Working Group provides advice on CADTH optimal use drug projects. Working group members also facilitate effective jurisdictional sharing of optimal use information.

Canadian Drug Expert Committee: The (CDEC) is a pan-Canadian advisory body to CADTH composed of individuals with expertise in drug therapy, drug evaluation, and drug utilization, and includes public members for a lay perspective. CDEC makes recommendations to each of the participating federal, provincial, and territorial publicly funded drug plans regarding the listings on their formularies. It also makes recommendations related to the identification, evaluation, and promotion of optimal drug prescribing and use in Canada.

Provincial Advisory Group: A Provincial Advisory Group (PAG) is in place to provide advice to the CADTH pCODR Advisory Committee (PAC) about operational issues, as well as to inform strategic and policy direction.

pCODR Advisory Committee: The CADTH pCODR Advisory Committee (PAC) provides strategic advice for pCODR's ongoing development and management, and provides advice on cancer-specific issues to ensure the pCODR program meets the needs of the provincial/territorial (and as of 2016, federal) governments and cancer agencies.

pCODR Expert Review Committee: The role of the pCODR Expert Review Committee (pERC) is to assess the clinical evidence and cost-effectiveness of cancer drugs in order to make recommendations to the provinces and territories and their respective cancer agencies (also includes the federal government as of 2016) to help guide their drug funding decisions. pERC includes patient members.

Health Technology Expert Review Panel: The HTERP is an advisory body to CADTH convened to develop recommendations on medical device health technologies to inform a range of stakeholders within the Canadian health care system.

Pharmaceutical Directors Forum: The members of the Pharmaceutical Directors Forum are representatives of the publicly funded drug plans. The role of this group is to share information on pharmaceutical policy and strategic initiatives; conduct intergovernmental dialogue regarding the delivery of drug benefit plans; collaborate on efforts to align pharmaceutical strategies and policies across Canada; and interact with other groups on pharmaceutical issues of common interest. CADTH provides secretariat support to the Pharmaceutical Directors Forum.

Policy Forum: The Policy Forum was created in response to the Health Technology Strategy 1.0, and the subsequent Implementation Strategy, approved by the Conference of Deputy Ministers in May 2004 and April 2005 respectively. The mandate of the Policy Forum is to provide federal, provincial, and territorial jurisdictions with opportunities to share information and collaborate on health technology policy initiatives, where it is beneficial to its members. Presently, the focus of Policy Forum activities is on issues related to the implementation, management, and decommissioning of medical device health technologies. CADTH provides secretariat support to the Policy Forum.

The Health Technology Analysis Exchange: The Exchange is a network of health technology assessment producers established in accordance with the Health Technology Strategy 1.0 and the subsequent Implementation Strategy, approved by the Conference of Deputy Ministers in May 2004 and April 2005 respectively. The mandate of the Exchange is to to provide a forum for HTA producers to share knowledge, information, and experience and to facilitate continuous quality improvement in the production and use of evidence-based information on health technologies. CADTH provides secretariat support to the Exchange.

Pan-Canadian Health Technology Assessment Collaborative: The Collaborative is a network of Canadian HTA producers and health decision makers with a focus on strategic alignment and joint initiatives. CADTH is a member of the Collaborative and also provides secretariat support.

Appendix 5 - Summary of Findings

Rating of Findings

Ratings have been provided to indicate the degree to which each evaluation issue and question have been addressed.

Relevance Rating Symbols and Significance:

A summary of Relevance ratings is presented in Table 1 below. A description of the Relevance Ratings Symbols and Significance can be found in the Legend.

Performance Rating Symbols and Significance:

A summary of Performance Ratings is presented in Table 2 below. A description of the Performance Ratings Symbols and Significance can be found in the Legend.