AI in regulatory: Experts say to look beyond validation

BARCELONA – As more regulatory professionals adopt artificial intelligence (AI) tools as part of their jobs, a panel of experts who spoke at the RAPS AI in Regulatory Conference advised attendees to take a risk-based, life-cycle approach. They also have detailed considerations for validating those tools and going further to ensure they can catch potential issues.

Leon Doorn, CEO of MedQAIR Services B.V., told attendees to expect notified bodies to take notice of their use of AI, as such tools are being used to generate information for quality management systems, set up reports, write technical documentation (TD), and even instruction manuals, which carries a risk of introducing errors. To address such risk, he said it is important to validate the AI systems and continuously monitor them.

Doorn said that while some regulatory professionals already use specialized AI tools, others use more general ones that serve more as AI assistants. He noted that the latter are better suited to tasks such as drafting and checking documentation, communication, and summarizing text. He also noted that a common issue is that people use AI tools on an ad hoc basis, which means they don't have sufficient records to validate their use.

On the other hand, Doorn said AI specifically designed for regulatory affairs may be better suited to some tasks, and while AI developers may do their own validation, users may need different AI tools to address separate functions. Ultimately, he said it's important to ask what the intended purpose of the AI tool is, consider its risks, test the tool, set controls, and monitor it.

Picking the right AI tool

Before picking a new AI tool, Simona Jerabkova, a director of regulatory affairs at Edwards Lifesciences, said that users need to ask themselves five important questions. First, they need to ask what the AI is intended to do and what kind of behavior is unacceptable. Secondly, what happens when it does something unacceptable, and who takes ownership of its mistakes? Next, users need to ask what needs to be validated in the AI, and what parts of the process require human oversight. Then it's important to figure out what triggers an AI revalidation. Finally, users need to ask whether the AI will improve the quality of their work or save significant time.

Jerabkova gave an example of how her company uses an AI assistant to submit a product application. She said they use the AI tool to retrieve documents from a library of predefined documents, after which the AI compiles a checklist, creates a draft, and summarizes that draft. Afterwards, the regulatory professional checks the AI's work for completeness and adds any missing details before finalizing the document.

"There is always a human in the loop, which is, in our case, at the end, where the regulatory affairs specialist reviews and finalizes the full dossier," said Jerbkova. "We are not relying fully on the AI."

When validating AI, Jerbkova said it's important to define the purpose of the tool, identify the risks and no-go zones. She noted that AI sometimes hallucinates to please the user, which means it is very important to calibrate the system during validation.

Validation resources

When validating AI tools, Doorn said there are several standards, regulations, and guidelines that companies can look to. He noted that the International Organization for Standardization's ISO 13485:2016 specifies requirements for how medical devices should meet Quality Management System (QMS) requirements, including validation before first use, revalidation after changes, ensuring risk proportionality, and record-keeping. He also noted that the US Food and Drug Administration’s (FDA) Quality Management System Regulation, which took effect earlier this year, is based on ISO 13485, and that the agency simultaneously issued final guidance on computer software assurance.

"What is nice in the new guidance set up at the FDA is that it really simplified the whole tool validation process," said Doorn. "They bring it down to a few simple steps."

Determining how much testing to do of the AI tool will depend on what the AI is being used for and how much risk is associated with that intended use, according to Doorn. For example, he said that low-process-risk AI that summarizes audit notes for human review, routes corrective actions and complaints, or suggests literature search terms may be sufficient with unscripted testing. High-process-risk AI tools that produce instructions for use, judge acceptability without human review, or collect essential device safety data may require scripted testing or scripted testing combined with unscripted testing.

Doorn said he has been involved in developing the International Electrotechnical Commission (IEC) IEC 63450 standard, which is expected to be published next year and is intended for the verification and testing of AI-enabled medical devices. While it isn't intended for AI validation, he said it includes a test for AI in which the final answer is unknown and can be useful for understanding how a user's AI tool is behaving.

Doorn warned that AI tools are often not up to date with the latest standards, and it's important to check which standards the AI is referencing, as that may get users in trouble. Ultimately, he said AI regulatory users need to remember that risk-proportionate AI tool validation is required under quality management systems, users should define the intended use of their AI per function and not per tool, they should use test methods that work without a fixed answer, and they should plan for model changes outside their control.

Beyond validation

Kerstin Kaspar, senior AI certification manager at TÜV AI.Lab, told the audience that, beyond validation, AI tools need to provide users with assurance that they will operate as intended, because they have specific risks and system complexities that require the validation process to be tailored to their intended use. She also noted that while testing the AI provides evidence of how it is operating, it still needs effective risk controls. Additionally, she said AI validation evidence and risk controls need to be traceable to their intended use, risks, and how the AI changes over its lifecycle.

"There are AI inherent risks that we really need to consider," said Kaspar. "The most known is bias, but we also have time-dependent risks and data risks and everything along the whole lifecycle.

"We want this validation to be across the whole lifecycle to ensure that our system is functioning properly," she added. "To do that, we cannot only rely on validation, we need to see what we need to control to have the whole functionality and to have the whole lifecycle covered."

Kaspar said that AI validation requires testing several key aspects of the tool, including whether the data used for the AI is appropriate, representative, and of sufficient quality. Furthermore, she said the testing should determine whether the AI system performs its intended task sufficiently, whether the results can be trusted, whether it performs robustly, whether it is secure, and whether its workflow performs as expected end-to-end.

Validation alone won't eliminate potential risks in the AI tool, according to Kaspar. She said there are four aspects of AI that need to be managed and controlled to minimize risk. Part of that includes evaluating whether the remaining risks after validation are acceptable for the product's intended use, and how changes and updates to the AI are controlled. She also highlighted the use of human oversight to reduce risk.

"Human oversight is super important, and it's a great tool that we have to double-check what the AI does," said Kaspar. "But what you need to consider is that human oversight needs to be set up properly, and the people that do the oversight need to really know what they are overseeing, how it works, what they are looking for."

Kaspar also added that human oversight isn't foolproof, and users still need to monitor and escalate issues if they find the AI is degrading or if other problems affect its accuracy.