Scientific Advisory Committee Digital Health Technologies,  June 9 and 11, 2021, summary of proceedings

On this page

• Attendance
• Welcome
• Chair's remarks
• Summary and general considerations
• Presentations
• Discussion
• Closing remarks

Attendance

Committee members

Core members: June 9 - Joseph Cafazzo (Chair), Aviv Gladman, Trevor Jamieson, Kendall Ho, Chris Kamel, Kim Hanson, Kumanan Wilson

June 11 - Joseph Cafazzo (Chair), Aviv Gladman, Trevor Jamieson, Kendall Ho, Kim Hanson, Kumanan Wilson

Ad hoc members: June 9 - Frank Rudzicz, Anna Goldenberg

June 11 - Anna Goldenberg

Presenters

Health Canada: Tyler Dumouchel

Other: Jaron Chong

Observers

Health Canada: June 9 - Patrick Assouad, Ian Glasgow, Justin Peterson, Daniel Yoon, Martina Buljan, Renate Kandler, Daniel Martire, Gregory Jackson, Tudor Fodor, Janet Hendry, Kinga Michno, Panyada Phandanouvong, Marc Lamoureux, Nicole Charron, Ursula Polack, Kenneth Joly, Miguel Martins, Grant Kealey, Kevin Day, Meghana Sharma, Syed Sibte Raza Abidi, Johanne Veenstra, Andrew Smith, Marc Lamoureux, Sven Schirmer, Emily Hollink, Sally Prawdzik, David Boudreau

June 11 - Patrick Assouad, Ian Glasgow, Justin Peterson, Daniel Yoon, Martina Buljan, Renate Kandler, Daniel Martire, Gregory Jackson, Tudor Fodor, Janet Hendry, Kinga Michno, Panyada Phandanouvong, Marc Lamoureux, Nicole Charron, Ursula Polack, Miguel Martins, Grant Kealey, Kevin Day, Meghana Sharma, Andrew Smith, Emily Hollink, Sally Prawdzik, Abdullah Farooqi, Robin Roberts, Tanya Ramsamy, Christine Leckie, Johanne Veenstra, Frederic Hamelin, Daniel Preece, Thomas Hazle, Tyler Dumouchel, Tamara Brown

Welcome

Kevin Day, Acting Executive Director, Bureau of Evaluation, Medical Devices Directorate (MDD), welcomed members to the committee's first virtual meeting. He gave an overview on how Health Canada has implemented innovative and agile regulatory measures to prioritize and expedite the regulatory review of in-demand COVID-19 digital health devices. He also provided a brief update on the Medical Devices Action Plan.

He then introduced the meeting topic, the regulation of unlocked artificial intelligence (AI) or machine learning-enabled medical devices as candidates for the advanced therapeutic products pathway. Advanced therapeutic products are novel products that challenge current regulatory frameworks for drugs, biologics or medical devices.

He thanked members for their time and participation.

Chair's remarks

Dr. Joseph Cafazzo, Chair, thanked members for participating in the meeting and confirmed acceptance of the agenda. He introduced the panel and gave an update on affiliations and interests declared by members. There were none that restricted SAC-DHT members from participating.

Summary and general considerations

Marc Lamoureux, Manager, Digital Health Division, MDD, provided a summary of previous advice from this committee used by Health Canada. For example, advice from previous meetings was incorporated into guidance to industry on cybersecurity of medical devices. In addition, committee feedback on training data, bias and generalizability was used to help Health Canada assess dozens of locked "machine learned" algorithms.

He also went over the meeting topics:

• scope of technologies being considered for regulatory measures
• new items that Health Canada should consider when assessing these products (for example, good machine learning practices)
• ways to enhance post-market monitoring to improve access while maintaining high standards of safety and effectiveness

Presentations

There were 3 presentations:

1. Ken Joly (Biologics and Radiopharmaceutical Drugs Directorate (BRDD), June 9:

• overview of regulatory innovation at Health Canada in the area of advanced therapeutic products

2. Tyler Dumouchel (MDD), June 9:

• detailed overview of Health Canada's current Medical Device Regulations
• the proposed advanced therapeutic products framework and regulatory considerations for unlocked machine learning-enabled medical devices (MLMDs)

3. Dr. Jaron Chong (Western University), June 11:

• overview of unlocked AI/MLMD from a clinician perspective

Presentation 1:

Mr. Joly said that agile regulations are needed for an evolving health landscape where products are becoming more complex and personalized (COVID-19 reinforced this need). Changes to the Food and Drugs Acthave been made to enable Health Canada to create regulatory measures to authorize advanced therapeutic products.

The use of tailored requirements will address a product's specific characteristics while maintaining Health Canada's high standards for patient safety. Modernized inspection powers will allow a more agile approach to compliance and enforcement, and a planned service will guide and assist innovators who develop potential advanced therapeutic products. As a single point of contact for external stakeholders, this service will:

• engage with innovators to stay current and anticipate regulatory needs
• facilitate discussions within Health Canada
• help innovators access relevant information

As part of an ongoing environmental scanning process, Health Canada will reach out to different types of stakeholders and identify potential advanced therapeutic products. Health Canada continues to engage with stakeholders to inform next steps and increase awareness of regulatory measures and the point-of-contact service.

Presentation 2:

Mr. Dumouchel clarified that machine learning is a subset of AI that enables systems to learn from data without being explicitly programmed. An MLMD is a medical device that uses machine learning, in part or in whole, to achieve the intended medical purpose.

Unlike locked MLMDs, unlocked MLMDs are expected to undergo pre-authorized changes (for example, to the model, algorithm, weights or parameters) following pre-market authorization. These pre-authorized changes will not be required to undergo additional pre-market evaluation before being implemented. The proposed advanced therapeutic products pathway may allow for more flexibility than current regulations. Manufacturers would be required to submit a plan for how the algorithm will learn and change while remaining safe and effective and ensuring quality.

Regulatory considerations around pre-authorized change plans include:

• balancing safety and effectiveness while facilitating market access to innovative products
• pre- and post-market requirements
• performance metrics
• interoperability
• verification and validation approaches
• applicability of the product to Canada's diverse population

The proposed advanced therapeutic products framework would follow a phased approach depending on clinical degree of autonomy, change effectuation, learning dynamics and change domain. This would allow Health Canada to optimize access while managing risk.

Presentation 3:

Dr. Chong reviewed the advantages and disadvantages of the 2019 US Food and Drug Administration's model on updates to AI/machine learning software. He also discussed the mechanisms and frequency by which Health Canada should receive software performance updates.

Dr. Chong then commented on:

• the feasibility of relying on the clinical community to provide meaningful data on performance
• how the clinical community might interface with the regulatory process for evaluating the real-world performance of AI/machine learning software

Dr. Chong highlighted the importance for users to be informed of model versions. He also emphasized the importance of following good machine learning practices, which helps with the ability to trace training and validation data and enable model backtracking and concurrency. There should be clear mechanisms and processes in place for reporting model errors. Feedback given to manufacturers about their models should be timely, direct and recorded permanently.

Discussion

Proposed phased approach

Health Canada's initial proposal for regulating unlocked machine learning software includes a phased approach. This approach starts with lower-risk scenarios (such as devices that assist but not replace clinical judgment, device changes that were approved by another regulatory agency) and is based on the following factors:

• clinical autonomy
• error correction
• learning dynamics
• change domain

Discussion centred on the advantages and disadvantages of this proposal, including its:

• complexities
• challenges
• opportunities
• access to innovation
• appropriate regulatory oversight
• capacity to address industry and clinical readiness

The committee also reviewed the factors that may mitigate or aggravate risk. The importance of having an appropriate amount of regulatory oversight without adding unnecessary burden to innovators was highlighted.

Factors for consideration included the following:

• risks around operator error
• risks of delaying a potentially beneficial product from entering the market
• determining liability
• mechanisms for identifying when software is not functioning as intended
• contexts in which AI will be used
• finding a balance between allowing timely software updates while preserving patient safety
• importance of post-market surveillance to ensure that a product's benefits outweigh any risks
• types of quality control information that should be required from manufacturers

The first phase of the proposed phased approach may not fully represent all of the emerging technologies. The committee was concerned that Health Canada may not fully appreciate the wide range of AI technologies. It's important for the regulations to be flexible and agile.

As a starting point, the committee recommended that Health Canada consider first:

• identifying any regulatory gaps in the areas of highest need
• using the existing regulatory framework for as many products as possible
• reserving the advanced therapeutic products pathway for exceptionally novel and unique technologies

Good machine learning practices, device pre-specifications, algorithm change protocols and data management

The committee discussed the:

• information that Health Canada should consider when assessing the manufacturer's:
• quality management system and good machine learning practices
• device pre-specifications
• algorithm change protocol
• software modifications that Health Canada should allow without needing another review
• software modifications that should trigger additional regulatory pre-market evaluation before being implemented

The committee noted the importance of facilitating beneficial updates with minimal requirements and conducting further review where results are less predictable. A great deal of innovation is coming from start-up companies. In light of this, restrictive regulations may disproportionately disadvantage these newer and smaller manufacturers who may not have the same infrastructure or capacity as large, established companies.

The committee discussed the need to engage appropriate medical expertise as part of the approval process.

Also, machine-learning enabled software cannot be expected to work in the same way in a variety of environments and must be updated, retrained and specialized for their specific context. The regulations should reflect this. The committee also discussed the extent to which user transparency should be incorporated.

Pre-specifications (such as changes to inputs, performance or intended use) are specific to an application and would be difficult to regulate broadly. A risk analysis for each product should be conducted. Health Canada should:

• require that manufacturers provide guidance on using the software in a clinical setting, maintaining the software and ensuring its performance for the safety of users and patients
• look at using neutral or third-party datasets for validating performance claims and providing better assurance of real-world performance

Post-market activities

In addition to mandatory incident reporting, the committee talked about the information that manufacturers should provide to Health Canada as part of their post-market activities. For example, information could include sales, usage, performance, user feedback and changes to performance or safety since product launch.

The committee talked about the feasibility of relying on clinical users to provide feedback on software performance. Feedback could be embedded in the software and would help to guide post-market enforcement actions.

Currently, the reporting of incidents with a medical device is required. The proposed advanced therapeutic products pathway plans to shift regulatory oversight toward more post-market surveillance. The committee said it may be difficult to regulate AI/machine learning technologies before knowing what type of information should be monitored and reported on an ongoing basis for every circumstance. Higher-risk technologies require more complex monitoring and safety parameters.

Health Canada should consider requiring manufacturers to create their own post-market monitoring strategies and explain why they are valid for their particular technology. Based on their experience and risk analysis, they should dictate what the range of acceptable performance is and set expectations for local users to monitor for deviations.

Incident reporting can be useful when users know what to look for and report on. Relying on clinical users to give their feedback on performance will be challenging. Users may not be able to identify the root cause of an issue when these algorithms are built into other tools.

Closing remarks

Kevin Day thanked committee members for their engagement and participation in the discussion. While regulating rapidly evolving AI/machine learning technologies continues to be a challenge, the committee's recommendations and deliberations provide valuable insight as to how Health Canada can facilitate access to these technologies.