Strategic performance measurement: Evaluating the effectiveness of risk management actions on toxic substances in protecting Canadians and their environment

Executive summary

The Government of Canada actively manages toxic substances through the Chemicals Management Plan. When a substance is found toxic to human health or the environment under the Canadian Environmental Protection Act, 1999 (CEPA), the Government takes action to manage the risk identified. While this work has been ongoing for many years, no systematic evaluation of these actions has been undertaken. The fall 2018 report of the Commissioner of the Environment and Sustainable Development (CESD) to the Parliament of Canada found that Health Canada and Environment and Climate Change Canada should do more to “assess whether they were meeting their overall objectives to reduce the risks of toxic substances to human health and the environment”. 

In response to the CESD recommendation, the following Performance Measurement Evaluation Strategy (the Strategy) was developed. The Strategy sets out the approach for evaluating the effectiveness of actions taken to address risks posed by substances found toxic under CEPA. It also outlines why, how and when performance measurement evaluations for the risk management of these substances will be undertaken. 

Each evaluation will compare the current state of a toxic substance with its state before risk management tools were implemented. Findings will be based on the best available information. However, evaluating how risk management strategies and tools impact human health and the environment is a lengthy process. For example, it can take decades to see a decrease in the presence of certain environmental contaminants in humans, such as lead.

Performance measurement evaluation reports will determine whether: the objectives have been met; progress can be measured but additional action is required; or, the risk management strategy or action has not achieved the intended results.

Follow-up actions will be recommended where objectives are not being met, or where linkages between actions and outcomes are not clear. Future evaluations may be recommended for some substances, while others may simply require routine monitoring. 

For substances requiring performance re-evaluation, plans to address gaps and track key indicators will be developed. Follow-up actions may include undertaking new research, adjusting the objectives of risk management approaches or instruments, and/or recommending substances for additional actions, such as monitoring, compliance promotion or enforcement. The readiness and prioritization exercises will be repeated to ensure that new information and emerging concerns are considered. Information will be available on the Performance Measurement Evaluation for Toxic Substances webpage, including performance measurement evaluation reports, work plans, and details on how to provide information. 

The performance measurement evaluation process will help ensure that risk management measures are applied where needed to protect Canadians and the environment from toxic substances. Through the evaluation of risk management actions, Canadians can have confidence that their health and the environment are being protected from harmful substances. 

Taking action on toxic substances: How it works in Canada

Chemicals are part of everyday life, essential to our economy, our communities and our homes. While chemicals provide benefits, they may also have harmful effects if not properly managed. The Canadian Environmental Protection Act, 1999 (CEPA) is one of the most important environmental laws in Canada to assess and manage substances^{Footnote 1}. A guiding principle of CEPA is to prevent pollution and manage the risks from potentially harmful substances to which humans and the environment might be exposed. CEPA describes the conditions under which a substance is "toxic" as those that enter or may enter the environment at levels or conditions that:

• have or may have a harmful effect on the environment and its biological diversity;
• are or could be dangerous to the environment on which life depends; or
• are or could be dangerous to human life or health.

The Government of Canada, through the Chemicals Management Plan, assesses substances to determine if they pose a risk to human health or the environment through the risk assessment process. When a risk is identified for a substance, the risk management process begins. Risk management strategies and tools are identified and implemented to mitigate the risks. By taking actions, including the development and implementation of risk management instruments, carrying out compliance promotion activities and enforcing regulations, the Government of Canada can help reduce the exposure of Canadians and their environment to toxic substances. The performance measurement evaluation process helps to evaluate the effectiveness of risk management. Figure 1 illustrates the role of performance measurement evaluation in the chemicals management cycle.

Risk management actions may involve using regulatory and non-regulatory tools under CEPA, the Canada Consumer Product Safety Act, the Food and Drugs Act, the Pest Control Products Act, the Fisheries Act, or other legislation. The objectives of risk management strategies or tools vary from decreasing the presence of a substance in consumer and commercial products; to minimizing the release of substances to the environment; to the complete prohibition and virtual elimination of a substance entering or being used in Canada. Tools can include, but are not limited to, regulations (e.g., regulations administered by Environment and Climate Change Canada and by Health Canada), Significant New Activity (SNAc) provisions, Pollution Prevention Planning Notices, codes of practice, release guidelines, environmental performance agreements, and outreach and engagement. 

Engaging stakeholders and the public is central to managing toxic substances in Canada. The public and stakeholders are engaged when a substance is assessed and proposed to be concluded as toxic to human health or the environment. Engagement continues throughout the risk management process, including identifying risk management options and approaches, selecting tools to mitigate risks, and developing or modifying those tools. Engaging relevant stakeholders during the performance measurement evaluation process is important to ensure all relevant data has been received and considered. Providing results to Canadians on how well risk management strategies and tools have performed will enable stakeholders and the public to better understand how effectively their health and the environment are protected from toxic substances. 

This Performance Measurement Evaluation Strategy outlines when and why performance measurement evaluations on the risk management of toxic substances will be undertaken and how the public and interested stakeholders can stay informed and provide information.          

1. Why measure performance?

Performance measurement evaluation is an essential part of the risk management cycle because it validates whether or not risk management strategies and tools are helping to protect Canadians and their environment. A performance measurement evaluation will demonstrate the progress these risk management strategies and tools have made toward meeting their individual objectives and if changes are needed over time. Under this Strategy, work plans and timelines will be developed to measure performance. Performance measurement will be considered early in the risk management process (e.g., at the risk management scope or approach stage) to better integrate objectives or targets, and a data collection plan will be developed to enable future performance measurement evaluation. Potential environmental and human health factors impacted by climate change and emerging use patterns of substances will also be considered, when appropriate, in the context of the performance of risk management tools. 

The Commissioner of the Environment and Sustainable Development (CESD) provided a report to the Parliament of Canada in fall 2018 on the management of toxic substances. It found that Health Canada and Environment and Climate Change Canada should do more to “assess whether they were meeting their overall objectives to reduce the risks of toxic substances to human health and the environment”. The report recommended that both departments “establish a long-term, systematic approach to evaluate how effective their actions are in controlling toxic substances”. In response to the CESD’s recommendation, this Performance Measurement Evaluation Strategy was developed. The Strategy sets out the approach to evaluate the effectiveness of actions taken on substances found toxic under CEPA.

Results from performance measurement evaluations will help demonstrate to Canadians how effectively risks from substances are being managed in Canada, where improvements can be made to risk management strategies and tools, and how the efforts to manage risks from substances help protect Canadians and their environment in the long term. It should be noted that there may be minor differences between how Health Canada and Environment and Climate Change Canada implement the Strategy, based on their specific priorities, activities, resources and processes.

2. Selecting substances for performance measurement evaluation

The performance measurement evaluation approach will focus on the substances that are deemed harmful to human health or the environment and meet the CEPA definition of a toxic substance. The selection process consists of two distinct exercises – readiness and prioritization.  A substance is “ready” for performance measurement if the necessary criteria to measure performance are met. For substances that are not yet ready, the missing elements will be identified and where appropriate, plans detailing what is needed to fill the missing elements will be developed. Substances that are ready will be identified for “prioritization”. The readiness and prioritization exercises will be repeated on a cyclical basis to ensure that new information and emerging concerns are considered.

Certain toxic substances are subject to ongoing performance measurement and reporting on progress under specific programs. Additional performance measurement activities for these substances beyond those already underway may not be necessary.^{Footnote 2}

2.1 Readiness

Some specific and minimum information must be available on a substance (identified below), to evaluate the performance of the corresponding risk management strategies and tools. The readiness step helps to identify which substances can proceed to prioritization for performance measurement, which substances are not ready, and the specific information required to make those substances ready to proceed. The following criteria will help determine which substances are ready for performance measurement evaluation:

• an environmental or human health objective or target has been identified;
• risk management tools have been implemented;
• sufficient time has passed since risk management tools were implemented for changes to be observed;
• key performance indicators have been identified and data are available; and
• baseline or trend information is available to assess key performance indicators.

Key performance indicator data are used to evaluate progress in meeting environmental and/or human health objectives. Available quantitative and/or qualitative performance indicators are considered for each substance. These can include, but are not limited to:

• concentrations and/or quantities in environmental media (e.g., water, sediment, air, soil) and food;
• concentrations in the human body, known as biomonitoring (e.g., in blood, urine, hair);
• concentrations in wildlife;
• concentrations in products available to consumers;
• quantities or concentrations released from sources (e.g., industrial facilities);
• number of products containing a substance currently available to consumers;
• quantity of a substance present in Canada, including in products;
• epidemiological data/disease clusters/incidence of disease;
• compliance with enforceable risk management tools; and/or
• industry participation in non-regulatory risk management tools (e.g., Codes of Practice).

2.1.1 Outcome of the readiness exercise

Substances that meet all the readiness criteria are then prioritized to determine the order in which those substances will be selected for performance measurement evaluation. Figure 2 below summarizes the readiness exercise. Readiness plans will be developed for substances that do not meet all the readiness criteria. Those plans will explain why a substance is not ready, and provide steps and timelines for achieving readiness. Readiness will be achieved easily in some cases, for example by establishing missing risk management objectives. In other cases, many years will be required to achieve readiness, such as in cases where analytical methodologies need to be developed, validated and implemented to obtain the key performance indicator data.

2.2 Prioritization

All substances that are identified as “ready” will be prioritized to establish the order in which they will be evaluated. Prioritization identifies which ready substances should be evaluated first so that any changes to risk management efforts can be directed where they are most needed.

2.2.1 Criteria

Prioritization is based on certain criteria including, but not limited to, and in no particular order:

• time passed since commitment to manage risks, where the longer a risk management strategy or tool has been in place, the more of a priority it may be;
• alignment with departmental activities that are planned or underway, that would inform or complement performance measurement evaluations;
• alignment with international priorities and activities (e.g., the introduction of new or modified regulations in other jurisdictions);
• completeness of datasets to enable a comprehensive performance measurement evaluation; and
• emerging concerns identified by indicator trends or findings (e.g., data indicating low compliance with a risk management tool, or increased uses or releases of a substance, may be considered a higher priority for evaluation).

2.2.2 Process

Prioritizing a substance is based on the number of criteria met, which criteria are met and the extent to which they are satisfied. For example:

• a substance meeting several criteria or satisfying more fully many criteria may be prioritized over other ready substances for earlier performance measurement evaluation; or
• a substance with a very complete data set across all key performance indicators and over the applicable timeline for a risk management strategy or tool may be rated more highly than a substance with a data set covering only a few years.

Some criteria may be given more weight when prioritizing a substance for evaluation, i.e., to reflect what is important for measuring performance and what is important for protecting human health and the environment. For example, a substance that has been risk managed for a long time may not be as high a priority as a substance that has been managed for less time but where there are emerging concerns identified through indicator trends or other sources of information. In that case, a substance risk managed for less time may qualify for earlier performance measurement than one that has been risk managed for a longer period, in order to evaluate the on-going effectiveness of the risk management strategies and tools and determine if adjustments are needed.

2.2.3 Outcome of the prioritization exercise

Prioritization determines the order that substances will be evaluated, with a rationale for their assigned level of priority. A work plan will be developed and periodically updated with associated timelines for prioritized substances. Timelines will be made public and accessible through the Performance Measurement Evaluation of Toxic Substances webpage. While the number of evaluations conducted may vary year to year, one to two evaluations are expected to be conducted at a minimum each year.

3. Measuring progress in protecting Canadians and the environment: Evidence-based evaluations

The performance of risk management strategies and tools will be evaluated by assessing all relevant available information, and comparing the current state with conditions before risk management tools were implemented (known as baseline data). Findings will be based on the best available information for key performance indicators. Where baseline data do not exist, other approaches will be used, including, but not limited to, guidelines and targets. Conclusions will be based on whether risk management strategies and tools met the environmental or human health objectives. While efforts will be made to identify links between the findings and the implementation of risk management strategies and tools, these linkages are challenging considering multiple external variables. The aim will be to identify links based on information available while acknowledging uncertainties. In addition, new information (e.g., emerging concerns or new sources of exposure) will be taken into consideration to ensure long-term effectiveness of actions in place. The evaluations will be comprehensive and evidence-based, and will determine if the environmental and/or human health and risk management objectives have been met. Performance measurement evaluation reports may include recommendations for follow-up actions.

3.1 Data Sources

Key performance indicator data will be collected and analyzed to evaluate the effectiveness of risk management tools that have been in place. Where possible, data will be sourced from programs and activities already in place. In other cases, data may need to be generated, as identified in the readiness or prioritization steps. Sources of data may include, but are not limited to:

• environmental monitoring: Chemicals Management Plan Environmental Monitoring and Surveillance Program, which leverages existing departmental programs such as the National Air Pollution Surveillance Program, Freshwater Quality Monitoring and Surveillance, Air Quality program science;
• biomonitoring: Canadian Health Measures Survey, Maternal-Infant Research on Environmental Chemicals Study;
• releases: National Pollutant Release Inventory, Air Pollutant Emission Inventory;
• food monitoring and surveillance: Canadian Total Diet Study and targeted studies;
• non-confidential information submitted by industry and other stakeholders: information provided through a voluntary or mandatory initiative, cosmetic notifications, Significant New Activity notifications;
• compliance monitoring and enforcement data: product surveillance, laboratory testing, threat risk assessments, results of departmental enforcement activities;
• information submitted in response to risk management instruments: Pollution Prevention Planning Notices, codes of practice, regulations, and environmental performance agreements; and
• other existing sources of relevant information: 
  • Canadian Environmental Sustainability Indicators;
  • Northern Contaminants Program; 
  • Other Government of Canada programs and departments; 
  • Other Canadian jurisdictions and levels of government; and
  • Academic research, international sources of data.

3.1.1 Stakeholder engagement

Evaluations will consider all relevant information that is readily available. However, the public and stakeholders may have information that is not available to the government. To address information needs, Health Canada and Environment and Climate Change Canada may contact specific stakeholders for additional details that would inform certain evaluations, including providing baseline information or obtaining more current data. The types of information that stakeholders may possess that could inform performance measurement evaluations could include, but are not limited to:

• information outside of reporting requirements such as the National Pollutant Release Inventory (NPRI), mandatory surveys or risk management tools;
• information on the concentration of a substance in a product of interest or a release to the environment;
• research activities related to a substance, including monitoring; and
• quantity of a substance throughout the supply chain.

Stakeholders who may have relevant information could include industry, non-governmental organizations, and other levels of government, such as provinces, territories or municipalities.

Stakeholders can submit data to inform upcoming performance measurement activities through the online reporting system on Environment and Climate Change Canada's Single Window. If there are questions regarding submissions, they can be addressed by Environment and Climate Change Canada’s Substances Management Information Line. Stakeholders who provide information may submit a request that it be treated as confidential. 

3.2 Analysis

Progress in achieving the human health and/or environmental objectives for a substance will be determined by analyzing the best available data. Analyses of specific data sets contribute to the overall evaluation findings for a substance. As part of the overall analysis, the evaluations will also look at what influence a risk management strategy or tool has had on any observed trends or changes. For example, if a reduction in emissions of a substance is measured following the implementation of risk management strategies or tools, these reductions could be attributed to the actions taken. This analysis can help determine the ongoing relevance and effectiveness of a strategy or tool in meeting their objectives.

While the analysis will be conducted with the best available information, it should be noted that it can take a significant amount of time to see how risk management strategies and tools impact human health and the environment. For example, it can take decades to see a decrease in the presence of certain environmental contaminants in humans, such as lead.  As a result, the timeline for conducting performance measurement evaluation can vary between substances.

4. Reporting: Sharing the progress in managing risks

Performance measurement evaluation reports will summarize relevant indicator data, analysis, and other findings to demonstrate how effectively risk management measures have met objectives. Performance measurement evaluations will make one or more statements on the overall progress achieved, such as:

• the environmental and/or human health objectives have been met or there is sufficient progress toward achieving objectives;
• progress has been made to meet objectives but would benefit from additional information to determine whether additional actions may be required for ongoing effectiveness;
• the risk management strategy or tool is not achieving the intended results and follow-up actions will be recommended; or
• the evaluation is not conclusive and additional time, actions and/or data are required to assess the overall risk management of a substance.

Follow-up actions will be recommended where the performance measurement evaluation shows objectives are not adequately being met, or where linkages between actions and outcomes are not clear. Future performance measurement evaluation may be recommended for some substances, while others may only require ongoing monitoring. For those substances requiring future re-evaluation, data collection or evaluation plans will be developed to address gaps and to track key performance indicators based on the current findings.

Follow-up actions could include, but are not limited to:

• recommending changes to the risk management instrument or development of a new instrument (e.g., regulations);
• working with partners to address data gaps;
• undertaking studies or research;
• recommending substances for other activities, such as Identification of Risk Assessment Priorities monitoring (e.g., new sources of exposure, emerging science), compliance promotion or enforcement.

Where the performance measurement evaluation shows objectives are adequately being met, risk management strategies and tools will be maintained, along with the monitoring of any new information as required. Lessons learned from the outcomes of performance measurement evaluation will be applied in the development of future risk management strategies and tools.

5. Conclusion: Building confidence with systematic evaluations

Performance measurement evaluation will be considered when developing a risk management strategy or approach. Baseline data important for future performance evaluations will be gathered and ongoing data collection plans will be developed so that the best possible information is available for future analysis.

Performance measurement findings may be used in risk management and risk assessment prioritization. Information related to risk management performance measurement will be available on the Performance Measurement Evaluation for Toxic Substances webpage, including performance measurement evaluation reports, work plans and details on how to provide information.

6. Glossary