Evaluation of the Public Health Agency of Canada's Blood Safety Contribution Program 2017-18 to 2021-22

• Prepared by the Office of Audit and Evaluation
• Health Canada and the Public Health Agency of Canada
• February 2023

Table of contents

• List of acronyms
• Executive summary
• Program description
• Evaluation approach
• Findings
• Conclusions
• Recommendations
• Management Response and Action Plan
• Appendix 1: Budget and actual expenditures
• Appendix 2: Approach and limitations
• Appendix 3: Canada's transfusion and transplantation system
• End notes

List of acronyms

BSCP

Blood Safety Contribution Program

CTO

Cells, Tissues and Organs

CTOSS

Cells, Tissues and Organs Surveillance System

PHAC

Public Health Agency of Canada

TESS

Transfusion Errors Surveillance System

TTISS

Transfusion Transmitted Injuries Surveillance System

Executive summary

This report presents findings from the evaluation of the Blood Safety Contribution Program (BSCP) for the period of 2017-18 to 2021-22. The purpose of the evaluation was to assess the impact, usefulness, and relevance of the information generated from three surveillance systems: the Transfusion Transmitted Injuries Surveillance System (TTISS), the Transfusion Errors Surveillance System (TESS), and the Cells, Tissues and Organs Surveillance System (CTOSS). This evaluation did not assess activities undertaken by the provinces and territories, nor their participation in the BSCP.

Program context

Blood safety was recognized as a major issue in Canada following the contamination of the Canadian blood supply with the Human Immunodeficiency Virus (HIV) and the Hepatitis C virus, starting in the late 1970s and continuing throughout the 1980s. In the final report of the Commission of Inquiry into the Blood System in Canada (also called the Krever Inquiry), issued in November 1997, Justice Krever emphasized the importance of surveillance for blood safety in Canada. The federal government's response included a series of initiatives to support and strengthen the safety of Canada's blood system, including the Blood Safety Contribution Program (BSCP).

The Blood Safety Contribution Program (BSCP) is intended to support the development and enhancement of provincial and territorial systems that monitor adverse events associated with blood transfusions and blood products, as well as cell, organ, and tissue transplantation. The BSCP provides direct funding to provinces and territories to carry out surveillance activities using the three systems (TESS, TTISS and CTOSS) to track the safety of blood and blood products, as well as cells, tissues, and organs (CTOs).

Key findings

As the need and use of blood, blood products, and CTOs in transfusion and transplantation activities continues to increase in Canada, there is an elevated risk of adverse events related to transfusion and transplantation. Monitoring adverse events will allow for a quicker reaction in the event of a new or previously unknown blood and CTO safety issue or threat.

Evidence collected highlighted the potential value of the three surveillance systems developed by PHAC as a result of the Krever Commission's report to the overall biovigilance system. Users of these systems felt that the right type of data is being collected, but they noted that PHAC was not sharing all the information it collected. They wanted information on emerging issues and threats across the country to the safety of blood, blood products, and CTOs communicated in a timely fashion, rather than a simple summary of national-level data to better inform planning and decision making within their respective jurisdictions.

Surveillance data generated by provinces and territories was appropriate and is being used within their respective jurisdictions. However, national information provided by PHAC is not timely for users and has not been used to further inform planning and decision making within provinces and territories. In some cases, provinces and territories are undertaking some level of multi-jurisdictional analysis themselves which they feel should be PHAC's role.

It should be noted that the timeliness and usefulness of surveillance information were also identified as issues during the 2014 BSCP evaluation, as well as the expectation that all three surveillance systems would be fully operational, as originally outlined by program authorities. Although the program stated that TESS and CTOSS transitioned to fully operational sentinel systems in 2009 and 2018, respectively, no clear evidence was provided of these decisions and that these were communicated to partners. In fact, according to the Agency's public website, CTOSS is still identified as a pilot project, whereas TESS is not clearly identified as a sentinel system, which is the case for other sentinel surveillance systems.

Recommendations

The evaluation evidence discussed in this report led to the development of the following recommendations.

Recommendation 1: Clearly define and communicate PHAC's role, responsibilities, and priorities, in collaboration with all partners, with respect to:

• financially supporting surveillance systems monitoring adverse events and errors associated with the transfusion of blood, blood products and cell, organ and tissue transplantation in Canada; and
• monitoring, analyzing and reporting adverse events and errors linked to transfusion and transplantation activities in Canada.

Recommendation 2: Based on the outcome of the first recommendation, determine the necessary structures to ensure the timely release of BSCP surveillance information.

Program description

Program context

Blood safety was recognized as a major issue in Canada following the contamination of the Canadian blood supply with the Human Immunodeficiency Virus (HIV) and the Hepatitis C virus in the late 1970s and continuing throughout the 1980s. In the final report of the Commission of Inquiry into the Blood System in Canada (also called the Krever Inquiry), issued in November 1997, Justice Krever emphasized the importance of surveillance for blood safety in Canada. The federal government's response included a series of initiatives to support and strengthen the safety of Canada's blood system, including the Blood Safety Contribution Program (BSCP).

Program profile

The BSCP provides funding, through contribution agreements, for provinces and territories to support the development and enhancement of provincial and territorial systems that monitor adverse events and errors associated with blood transfusions and blood products, as well as cell, organ, and tissue (CTO) transplantation. BSCP funding can also be used for targeted research to identify and assess risks associated with the use of blood, blood products, cells, organs, and tissues. BSCP funding recipients include provincial and territorial governments, transfusion and transplantation centres and agencies and groups designated by provincial and territorial Ministries of Health to undertake surveillance of adverse events and errors related to blood, tissues, and organs, as well as Canadian not-for-profit organizations that support transfusion adverse event and error surveillance in the provinces and territories.

BSCP beneficiaries include funding recipients who benefit from improved transfusion and transplantation knowledge and increased capacity to collect and analyze data. When BSCP was put in place in the 1990s, public health practitioners were expected to benefit from improved knowledge and faster diagnosis of infectious diseases related to blood, blood products, and CTO transplants. Federal and provincial health authorities were expected to benefit from improved availability of evidence to inform health policy development, public health planning, program development, and decisions on actions to respond to outbreaks or other urgent public health events. Ultimately, Canadians were expected to benefit from higher quality blood, blood products, and CTO transplant products, as well as improved practices leading to fewer adverse events and improved patient safety.

The BSCP includes the following surveillance systems:

• Transfusion Errors Surveillance System (TESS): TESS is a non-punitive, anonymous, web-based error reporting system designed to capture actual events that may or may not have resulted in adverse consequences in the transfusion chain for blood, blood components, and plasma derivatives. It also includes "near-miss" events, where an error is caught before a product is administered to a patient. TESS is a pilot project started in 2005 and involving 15 hospitals in four provinces.
• Transfusion Transmitted Injuries Surveillance System (TTISS): TTISS is a national voluntary surveillance system in place since 2001 that captures moderate and serious adverse events related to the transfusion of blood and blood components, and plasma derivatives. Events are captured using data provided by hospitals, as well as provinces and territories. TTISS tracks twenty-four adverse events and infections. All provinces and territories, with the exception of Nunavut, provide data to the TTISS. Currently, it monitors over 94% of blood transfusion activity in Canada. Hospitals report adverse transfusion events (ATEs) to provincial and territorial surveillance offices using a standard format. Provincial and territorial surveillance offices then submit the data to PHAC.
• Cells, Tissues and Organs Surveillance System (CTOSS): CTOSS is a surveillance system (since 2011) involving five transplantation sites within Canada initially focusing on the surveillance of recipients of tissue allografts and tissue products; it currently does not track information about organs and cells. If it were to become fully developed, the system would collect data on moderate and severe adverse events, eventually expanding to include serious errors and near misses, for the purpose of monitoring trends in known and emerging risks, and reducing the transmission of infectious diseases due to transplantation.

Recipients receive BSCP funding to establish activities to monitor adverse events and errors associated with the transfusion of blood, blood products, and CTO transplantation that could include infectious diseases, allergic, and immune-mediated events (i.e., to establish the three surveillance systems). They develop and enhance surveillance and support targeted research activities to identify and define risks associated with the use of blood, blood products, and CTOs.

Recipients then submit data^{Footnote 1} to PHAC based on an agreed-upon data sharing arrangement; the approval of which usually occurs annually between May and July. Program staff then assess and validate the data and reconcile it with data in other reporting systems, notably mandatory reporting to Health Canada to ensure completeness. Once the data have been validated, program staff analyze the data and discuss the results with the provinces and territories at meetings, conferences, or via webinars. The final step in the process is to produce a formal report that is sent back to recipients for review and comment before it is eventually released to the public.

An annual budget of $4 million is allocated to the BSCP, including $2.19 million in contributions, as well as $1.6 million in salaries, operations, and maintenance.

Evaluation approach

Evaluation scope

The evaluation focused primarily on the value of PHAC's Blood Safety Contribution Program (BSCP) and covered activities from 2017-18 to 2021-22.

The evaluation used multiple lines of evidence, both qualitative and quantitative, to ensure triangulation of findings. These included a literature and document review, key informant interviews, and a comparison of surveillance systems from other countries. See Appendix 2 for detailed methodology, limitations, and mitigation strategies).

Evaluation questions

Attention was given to the impact of program activities, the timeliness and usefulness of the three surveillance systems, whether the right components were in place to deliver the BSCP, and whether it has remained relevant over time.

1. To what extent is PHAC's surveillance information being used to guide public health action related to the safety of blood, blood products and CTOs?
2. How are surveillance data being collected, disseminated and used to support transfer of knowledge, expertise and best practices?
3. Are the required program components present, and are the right data being collected through PHAC's surveillance systems?
4. How has the program remained relevant over time in ensuring the safety and quality of blood and blood products?

Findings

PHAC's role in Canada's transfusion and transplantation system

Transfusion and transplantation adverse events pose a risk to the health and safety of Canadians. Although it is important that provinces and territories have the capacity to collect their own data to monitor and address adverse events from transfusion and transplantation activities, it is unclear what PHAC's role should be moving forward, including whether PHAC should continue to financially support the three transfusion and transplantation safety surveillance systems.

Evidence collected found that a national biovigilance^{Footnote 2} system to monitor transfusion and transplantation activities in Canada is important because:

• the number of transfusions and transplantations is increasing over time;
• Canada is one of the largest per capita users of plasma protein products in the world;
• the need for transplants is expected to increase by 152% over the next 20 years; and
• the risks to blood and tissue safety are increasing due to globalization, immigration, international travel, and climate change.

A national biovigilance system would mean that Canada could react in the event of a new or previously unknown blood and CTO safety risks. As the previous evaluation highlighted in 2014, since the 1996 Krever Commission, other organizations such as Canadian Blood Services (CBS), Hema-Québéc (HQ), and Health Canada have also established roles in Canada's transfusion and transplantation system to monitor blood safety activities and take corrective action to ensure product safety, quality, and use. Health Canada regulates the safety of blood and plasma-based products through the Blood Regulations. CBS and HQ work to ensure patients have reliable access to safe, high-quality blood, plasma, stem cells, and organs and tissues. PHAC monitors national-level transfusion and transplantation adverse events and errors from blood, blood products, and CTOs. Appendix 3 presents a summary of roles of key players related to the transfusion and transplantation system in Canada.

External stakeholders also mentioned the importance of the role that PHAC has played and specifically highlighted successes in terms of the following:

• establishing national tools and standards for surveillance (e.g., hospitals use a consistent reporting form and surveillance manual for data collection), although no examples of new tools or changes in standards were developed during the current evaluation period;
• convening standing committees and working groups to support the establishment, monitoring, and ongoing use of the surveillance systems;
• supporting knowledge translation activities that are geared towards reducing transfusion and transplantation health risks (e.g., monthly teleconferences to establish consistency in coding); and
• coordinating ad hoc committees to discuss national surveillance data.

Evidence highlighted how important it was that provinces and territories have the capacity to collect their own data to monitor and address adverse events from transfusion and transplantation activities. However, questions were raised about whether it was still appropriate for PHAC to financially support provinces and territories through contribution agreements to develop capacity related to surveillance systems.

Evidence collected showed that the TTISS is considered to be fully operational. Although the program stated that TESS and CTOSS transitioned to fully operational sentinel systems in 2009 and 2018, respectively, no clear evidence was provided of the decision to transition these two systems from pilot projects to sentinel surveillance systems and that this was communicated to partners. In fact, according to the Agency's public website, the CTOSS surveillance system is still referred to as a pilot project, whereas the TESS surveillance system is not clearly identified as a sentinel system (which is the case for other sentinel surveillance systems such as FoodNet Canada and the Canadian Nosocomial Infection Surveillance Program).

Originally, program authorities^{Footnote 3} were established to provide financial assistance to designated transfusion and transplantation centres in the provinces and territories to build the necessary capacity to effectively monitor these activities. These authorities identified the development of fully operational surveillance systems (i.e., TESS, TTISS, and CTOSS) in each province and territory as key results expected in the short term, although a specific timeline was not identified.

Table 1 shows the current national coverage for all three surveillance systems. In the case of TESS and CTOSS, only a few provinces and territories are currently participating in the surveillance systems. Interviewees flagged that delays to these two pilot surveillance systems are compromising the quality and completeness of the data these systems are collecting and their usefulness. It should also be noted that this data is currently not being captured by other partners, such as Health Canada, CBS, or HQ, thereby creating a gap in the transfusion and transplantation system in Canada.

Table 1: BSCP coverage across provinces and territories

Province/Territory	CTOSS	TESS	TTISS
Newfoundland	No	No	Yes
Prince Edward Island	No	No	Yes
Nova Scotia	Yes	Yes	Yes
New Brunswick	Yes	No	Yes
Quebec	Yes	Yes	Yes
Ontario	Yes	Yes	Yes
Manitoba	No	No	Yes
Saskatchewan	No	No	Yes
Alberta	Yes	No	Yes
British Columbia	No	Yes	Yes
Yukon	No	No	Yes
Northwest Territories	No	No	Yes
Nunavut	No	No	No

Supporting the transfer of knowledge, expertise and best practices

External interviewees highlighted that the annual working group meetings for the three surveillance systems are both valuable and useful. A review of governance documentation showed that the working groups for each surveillance system had met and discussed areas such as best practices, including updating the respective surveillance system's user manual (e.g., new reporting codes, new case definitions); exchanging updates (e.g., new training) and preliminary surveillance results, and challenges such as data quality issues (e.g., duplicates, missing data) or under-reporting in different sites. This was corroborated by external interviewees, who mentioned that these meetings were a good place to share information with others on what was working well, as well as identifying common concerns.

The use of surveillance information to guide public health action

National surveillance information has not been published in a timely manner. Furthermore, although the correct type of data is being collected, surveillance information released to users does not include all collected information, and only provides a summary of national data. As a result, surveillance information from all three systems is not currently being used to influence policy and decision making on emerging issues and threats to the safety of transfusion and transplantation of blood, blood products, and CTOs.

Timeliness of surveillance information

National surveillance information has not been shared in a timely manner, such as the release of national-level surveillance reports and infographics. Specifically:

• TESS: The most recent TESS report, which presents 2012-16 surveillance information, was published in March 2021. Additionally, an infographic providing 2017-19 information was shared with stakeholders in 2022.
• TTISS: Although the most recent TTISS surveillance information published online is from 2013, an additional report providing 2013-17 information was shared with stakeholders in early 2021.
• CTOSS: Although the most recent CTOSS surveillance information published online is from 2013, an infographic with data from 2018-19 was shared with stakeholders in mid-2021. Finally, 2011-17 CTOSS surveillance data was shared with users during a presentation to the CTOSS working group in October 2018.

This finding is similar to the 2014 BSCP Evaluation, which at that time noted that surveillance data had not been published since 2005. Service standards were developed in response to the findings of the previous evaluation, and committed to having a full surveillance system report distributed annually. However, no evidence was found that these timelines were implemented as planned.

In comparison, a review of other countries' surveillance systems found that the U.S. and U.K. have similar data from 2020 online, while Australia and the Netherlands have 2019 data available.

Information was shared with partners. The BSCP distributed infographics, which are high-level summaries of partially collected data from the surveillance systems. These documents were shared via email, and while this allows for quicker dissemination of information to users when compared to the production and online publication of full surveillance reports, these infographics were not distributed in a timely fashion. For example, infographics for TESS (2017-19), and CTOSS (2018-19) were only distributed in 2021.

All interviewees agreed that national surveillance information was not produced by PHAC in a timely manner, and, as a result, the information has become increasingly less useful and relevant as time passed. PHAC's timeline for dissemination is a stark contrast to the provincial-level timelines. Some provinces are able to produce their reports on an annual basis. For example, in Ontario, monthly newsletters (comparable to PHAC's infographics), are disseminated to hospital staff and stakeholders (e.g., Canadian Blood Services, the Ontario Regional Blood Coordinating Network, the Public Health Agency of Canada, Health Canada) to inform current trends in transfusion reaction reporting, and encourage hospitals to report in "real time."

Data sharing agreements outline different schedules for sharing data between PHAC and individual provinces and territories. Generally, provinces and territories share data with PHAC between May and July on an annual basis. It was not clear what impact, if any, these issues had on the timeliness of releasing surveillance information to users.

Usefulness of surveillance information

Provinces and territories are using their surveillance data from all three systems to guide their transfusion and transplantation practices to improve patient safety. For example, provinces and territories have:

• used data to estimate the risk associated with blood transfusion within their jurisdictions (in the case of TTISS);
• developed educational materials (including videos) for health care professionals (e.g., nursing and laboratory staff), as well as guidelines and manuals for practice and communication with staff;
• shared surveillance data with hospital staff in a timely manner (e.g., surveillance data dissemination via a monthly newsletter);
• shared information with other provinces and territories (e.g., to identify adverse transfusion events related to blood products);
• developed applications accessible from mobile devices (as well as web-based) for the recognition and reporting of signs and symptoms of adverse events;
• used data to influence legislation (e.g., in 2018-19, CTOSS data was used in New Brunswick to inform a new legislation on dental tissue surveillance); and
• used data for research (e.g., a 2018 peer-reviewed paper published the national TESS data). ^{Footnote 4}

The data that is currently tracked in the three surveillance systems is considered to be of adequate breadth and completeness, but only a subsection actually informs national information presented in dissemination products (e.g., surveillance reports, infographics) which are shared with users, thus limiting their usefulness.

Although external interviewees, including users, felt that PHAC collected the right type of data, they also wanted more detailed analyses of national data and trends, such as the identification of emerging issues and threats, and communication of these trends in a timely fashion. They noted the high potential impact that surveillance information could have if the data was timely and more thoroughly analyzed. They indicated that they would be more likely to use this data for their own planning and policy decision-making purposes if these concerns were adequately addressed in the future. Currently, provinces and territories focus on their own data and have also worked with their counterparts to discuss findings of note, and to inquire whether any identified issues are also present in other jurisdictions (i.e., provinces are doing their own multi-jurisdictional analysis).

All external interviewees noted how valuable PHAC's databases from the three surveillance systems could be to provinces and territories, federal partners, including Health Canada, and researchers, if they could have access to all of the information collected and contained in the databases to reconcile with their own information or to inform the development of research papers and guidelines in as close to real-time as possible. This also included a suggestion to provide access to partial data on a quarterly basis to better track and monitor potential trends.

Finally, external interviewees also noted that, although it is important and beneficial to have epidemiologists and scientists involved in the analysis of surveillance data, consulting external transfusion specialists and physicians would bring the different perspective of working directly in the public health environment to support program analysis. They also mentioned the need to work with these individuals to update national adverse reaction definitions (e.g., anaphylaxis, Transfusion-associated circulatory overload, transfusion-related acute lung injury) to help ensure accurate and consistent reporting across multiple jurisdictions. In particular, several external interviewees noted the need for the TESS manual to be updated to ensure a common understanding of these issues.

Conclusions

As the need and use of blood, blood products, and CTOs in transfusion and transplantation activities continues to increase in Canada, there is an elevated risk of adverse events related to transfusion and transplantation. Monitoring adverse events will allow for a quicker reaction in the event of a new or previously unknown blood and CTO safety issue or threat.

Evidence collected highlighted the potential value of the three surveillance systems developed by PHAC as a result of the Krever Commission's report to the overall biovigilance system. Users of these systems felt that the right type of data is being collected, but they noted that PHAC was not sharing all the information it collected. They wanted information on emerging issues and threats across the country to the safety of blood, blood products, and CTOs communicated in a timely fashion, rather than a simple summary of national-level data to better inform planning and decision making within their respective jurisdictions.

Surveillance data generated by provinces and territories was appropriate and is being used within their respective jurisdictions. However, national information provided by PHAC is not timely for users and has not been used to further inform planning and decision making within provinces and territories. In some cases, provinces and territories are undertaking some level of multi-jurisdictional analysis themselves which they feel should be PHAC's role.

It should be noted that the timeliness and usefulness of surveillance information were also identified as issues during the 2014 BSCP evaluation, as well as the expectation that all three surveillance systems would be fully operational, as originally outlined by program authorities. Although the program stated that TESS and CTOSS transitioned to fully operational sentinel systems in 2009 and 2018, respectively, no clear evidence was provided of these decisions and that these were communicated to partners. In fact, according to the Agency's public website, CTOSS is still identified as a pilot project, whereas TESS is not clearly identified as a sentinel system (which is the case for other sentinel surveillance systems).

Recommendations

The evaluation evidence discussed in this report led to the identification of the following recommendations.

Recommendation 1

Clearly define and communicate PHAC's role, responsibilities, and priorities, in collaboration with all partners, with respect to:

• financially supporting surveillance systems monitoring adverse events and errors associated with the transfusion of blood, blood products and cell, organ and tissue transplantation in Canada; and
• monitoring, analyzing, and reporting adverse events and errors linked to transfusion and transplantation activities in Canada.

The evaluation found that there is uncertainty as to whether PHAC should continue to financially support the three surveillance systems, given the original objectives of the BSCP. As mentioned in the previous evaluation, The BSCP program was intended to support the development and enhancement of provincial and territorial systems to monitor adverse events associated with the transfusion of blood, blood products and cell, organ and tissue transplantation. PHAC should therefore define its role, responsibilities, and priorities moving forward, including whether it should continue to financially maintain the three surveillance systems.

The Krever Inquiry emphasized the importance of surveillance for blood safety in Canada, a role that PHAC has tried to take up through its activities. However, the evaluation found that PHAC has not met the information needs of users. As a result, the surveillance information PHAC develops is not currently being used to inform policy and decision making related to transfusion and transplantation safety. In some cases, provinces and territories are undertaking some level of multi-jurisdictional analysis themselves, thereby filling this gap. PHAC should therefore review whether its current role is still appropriate in the current pan-Canadian landscape for monitoring adverse events associated with the transfusion of blood, blood products, and CTO transplantation in Canada.

If it is decided that PHAC has a role to play, it should work with users to better identify and address their needs, providing the level of detail necessary to inform policy and decision making to ensure the safety of blood, blood products, and CTOs in Canada. This should then be clearly communicated internally and externally, especially with provincial and territorial stakeholders, Health Portfolio partners involved in blood safety activities, and other stakeholders who may use the information stemming from the three surveillance systems.

Recommendation 2

Based on the outcome of the first recommendation, determine the necessary structures to ensure the timely release of BSCP surveillance information.

Concerns were also raised about the timeliness of the surveillance information PHAC produces and shares with users, which has had a negative impact on identifying emerging issues and threats to the safety of blood, blood products, and CTOs.

PHAC should therefore review and revise current internal surveillance activities to ensure surveillance information is shared with users in a timely fashion.

Management Response and Action Plan

Recommendation 1

Clearly define and communicate PHAC's role, responsibilities, and priorities, in collaboration with all partners, with respect to:

• financially supporting surveillance systems monitoring adverse events and errors associated with the transfusion of blood, blood products and cell, organ and tissue transplantation in Canada; and
• monitoring, analyzing and reporting adverse events and errors linked to transfusion and transplantation activities in Canada.

Management response:
Management agreed with the recommendation.

Action plan for recommendation 1

Action plan	Deliverables	Expected completion date	Accountability	Resources
CCDIC will work internally with its Federal partners to determine and define PHAC's roles and responsibilities. Once completed, CCDIC will coordinate consultations with blood and cells, tissues and organs (CTO) external partners to engage on the decision made.	Terms of Reference for the consultations (internal and external). Report with options clearly identifying PHAC's roles, responsibilities and priorities in blood and CTO safety surveillance. Report with options on the best vehicles and approaches for PHAC to financially support blood and CTO surveillance in Canada.	June 2024	DG/CCDIC VP, Infection Disease Programs Branch (IDPB)	Complete with existing resources
Action plan: Action(s) program management will take to address the recommendation Accountability: Senior Management and Executive (DG and ADM level) accountable for the implementation of each deliverable Resources: The human and/or financial resources required to complete recommendation, including the source of resources (additional vs. existing budget)

Recommendation 2

Based on the outcome of the first recommendation, determine the necessary structures to ensure the timely release of BSCP surveillance information.

Management response:
Management agreed with the recommendation, with conditions. The response will be guided by the outcome of the first recommendation. Details to be determined.

Action plan for recommendation 2

Action plan	Deliverables	Expected completion date	Accountability	Resources
CCDIC will work internally with its Federal partners to determine and define PHAC's roles and responsibilities. Once completed, CCDIC will coordinate consultations with blood and cells, tissues, and organs (CTO) external partners to engage on the decision made.	TBD (based on Recommendation 1)	TBD	TBD	TBD
Action plan: Action(s) program management will take to address the recommendation Accountability: Senior Management and Executive (DG and ADM level) accountable for the implementation of each deliverable Resources: The human and/or financial resources required to complete recommendation, including the source of resources (additional vs. existing budget)

Appendix 1: Budget and actual expenditures

The table below outlines budgets and expenditures for grants and contributions on the Blood Safety Contribution Program's activities to support and conduct surveillance across Canada. The variance between budgetary allotments and expenditures is also provided. This information is divided by fiscal years between 2017-18 and 2021-22.

Table 2: Budget and expenditures

Expenditures	Budget ($)	Actual ($)	Variance ($)	% Budget spent
	Grants & Contributions (G&Cs)	G&Cs
2017-18	$2,190,000	$1,711,430	$478,570	78.15%
2018-19	$2,190,000	$1,585,503	$604,497	72.40%
2019-20	$2,190,000	$1,616,995	$573,005	73.84%
2020-21	$2,190,000	$1,460,998	$729,002	66.71%
2021-22	$2,190,000	$1,792,157	$397,843	81.83%
Total	$8,760,000	$5,923,419	$2,782,917	78.15%

Appendix 2: Approach and limitations

Approach

Data was collected and analyzed through various methods and lines of evidence, as outlined below. These lines of evidence were also analyzed by triangulation to improve the reliability and credibility of evaluation findings and conclusions.

The evaluation examined the following key questions:

1. To what extent is PHAC's surveillance information being used to guide public health action related to the safety of blood, blood products, and CTOs?
2. How is surveillance data being collected, disseminated, and used to support transfer of knowledge, expertise, and best practices?
3. Are the required program components present, and is the right data being collected through PHAC's surveillance systems?
4. How has the Program remained relevant over time in ensuring the safety and quality of blood and blood products?

Methodology

Data for this engagement was collected using the following methods:

Interviews

Conducted interviews with 18 representatives from the following groups:

• 6 PHAC staff and management;
• 6 representatives from other federal government departments; and
• 6 provincial or territorial representatives.

Document review

Reviewed approximately 90 documents and files, including administrative files, contribution agreements, records of decisions, briefing materials, and surveillance reports and infographics.

Data analysis

Examined available program performance information, including published and unpublished surveillance data, and funded project reports.

Financial data review

Examined the financial costs of delivering activities at IDPB.

Comparative analysis

Reviewed the transfusion and transplantation surveillance activities in four other countries (the United States, Australia, United Kingdom, and the Netherlands).

Limitations

The following table outlines the limitations encountered during the implementation of the data collection methods selected for this evaluation.

Also noted are the mitigation strategies implemented to ensure that evaluation findings could be used with confidence in guiding program planning and decision making.

Limitation	Impact	Mitigation strategy
Interviews with internal staff and external stakeholders are retrospective in nature. This may lead to the provision of recent perspectives on past events.	This can affect the validity of assessing activities or results relating to improvements in the program area.	Triangulation of other lines of evidence was used to substantiate or provide further information on data received through the interviews.
Performance measurement and surveillance data were very limited.	As a result, the assessment of the BSCP's contribution to the safety of transfusion and transplantation activities in Canada relied heavily on key informant interviews and program documentation, where available.	Triangulation of other lines of evidence was used to substantiate or provide further information (including external perspectives) on performance measurement and surveillance data received from the Program.
Complete planned and actual financial data for the program was not available. Only planned and actual contribution expenditures were provided between 2017-18 and 2021-22.	There is a limited ability to assess efficiency quantitatively to support the discussion of value of PHAC's activities.	Triangulation of other lines of evidence was used to substantiate or provide further information on program value.
There were issues identifying current users of surveillance information to get their perspective on the value of PHAC's activities.	In certain cases, interviewees noted what value their input could provide to the evaluation due to their limited knowledge or interaction.	The evaluation relied on the snowball method to identify additional, appropriate interviewees who could speak to program activities and objectives.

Appendix 3: Canada's transfusion and transplantation system