BARCELONA – A panel of experts at the RAPS AI in Regulatory Conference discussed how artificial intelligence (AI) is being used by a wide range of stakeholders, including notified bodies and healthcare product manufacturers.
The experts agreed that it’s important for regulatory professionals to avoid complacency, stay up to date and in command of the technology, while also questioning its use.
Sebastian Fischer, regulatory strategy principal at TÜV SÜD, discussed how notified bodies were using AI to conduct conformity assessments. He highlighted a white paper produced by the German association of medical device notified bodies, Interessengemeinschaft der Benannten Stellen für Medizinprodukte (IG-NB), entitled "Use of Artificial Intelligence in the context of assessment procedures by Notified Bodies." The paper serves as a foundational document for notified bodies and outlines several general principles for using AI systems, including ensuring transparency and traceability, ensuring that conformity assessments meet legal and regulatory requirements, assigning responsibilities and liabilities, and more.
Fischer explained that TÜV SÜD has been using an AI system called the Intelligent Document Analysis Solution (IDAS) to assist with their work. He noted that it is used to conduct completeness checks on product dossiers to verify that they include all necessary technical documentation (TD) components, and that it can perform track changes so assessors can instantly see the differences between versions of documents. He also said the system keeps an accurate history of the assessment process, shares customer responses among assessors through document commenting, can locate all instances of identical or similar information in the TD, and can perform reference recognition to link parts of the TD to relevant standards.
"The idea is really to have a [large language model (LLM)] trained specifically with TD data information with the consent of a handful of our customers and have basic but powerful options within that tool to assist the assessors," said Fischer. "Because you have to make the data somehow accessible to the AI, it's usually quite convenient also to have a powerful search tool embedded... this is very convenient if you have a long TD and are looking for specific information in some place and saving time.
"Likewise, simple functions like identification of similar text blocks within the documents and comparing them, whether for consistency, completeness checks after submission, all these kinds of functionalities are there," he added.
Fischer highlighted that IDAS was also built with protection of confidential information in mind. He noted that the IDAS data is encrypted and securely maintained on TÜV SÜD's premises. He also emphasized that the data complies with the data protection requirements laid out in the EU General Data Protection Regulation (GDPR), and that the system is reset after an assessment is completed to ensure controlled amnesia.
Sandy Wright, head of devices at Scarlet, discussed how his company has been approaching AI for conformity assessments. Scarlet is a medical device notified body focused on AI medical devices and software as a medical device (SaMD). He noted that Scarlet is building its own software with an AI-first approach to the conformity assessment process.
In developing their AI system, Wright said they first break down the conformity assessment tasks they want the software to perform, build specific tools to assist with those tasks, validate them, and then iterate the whole process. He said that when they come up with tasks, they first build the context for the task, find evidence to support it, and then evaluate the task requirements. Next, he said they draft their findings and compile a report on the AI's performance.
"One thing we benefit from at Scarlet... is in-house expertise in this sort of technological domain, and this idea that we can combine the domain expertise of our internal clinicians, product reviewers, and auditors, with software and [machine learning (ML)] engineering capacity in order to develop these kinds of innovative tools that I think will be the future of conformity assessment," he said.
For example, Wright said that Scarlet has a tool called Find Evidence that can sift through large documents and present sections to the assessor that it deems relevant and important for evaluating the product. He said that Find Evidence is only one of the tools they have implemented to speed up the time-consuming evidence-finding part of a conformity assessment.
Claudia Ben Dov, CEO of InnHealthium, talked about how AI has been used in her consulting work with manufacturers to help solve issues such as identifying the early state of the art. She noted that one of the main issues that manufacturers face is the scattered evidence and unclear benchmarks, which may lead to a weak intended-purpose argument for their product.
Ben Dov said that what typically takes weeks of sifting through the available literature and various databases to present assessors with the best intended purpose can be done by AI in hours. She said the use of AI can help manufacturers de-risk their products early, but they also risk potentially missing important studies and documents that may have been useful.
Another problem where AI can help, according to Ben Dov, is identifying design inputs and use-related risks. She said that, again, AI can drastically speed up the process of drafting measurable inputs from intended purposes, closing information gaps and addressing potential conflicts. However, she said that AI users also risk the software inventing limits and overlooking harm pathways.
Ben Dov also said AI can be very helpful in developing software specifications and test cases, as well as addressing quality management system issues and TD audits.
Gert Bos, chief scientific officer at QServe Group, also spoke on the panel and noted that he and his company use AI daily. However, he asked the audience to consider the limitations of AI. He also raised concerns about how people may potentially abuse AI, such as using it for hacking and other nefarious purposes.
"A lot of the things that we have been discussing and should be discussing is how can we ensure that we treat [AI] as a model that needs to be trained like a kid that is very eager to learn, but we control the boundaries," said Bos. "How can we keep these boundaries alive?"
Bos said that others at the conference have reiterated that responsibility ultimately lies with people, and he asked the audience to consider how companies should be organized to keep humans in command of AI. He also highlighted potential overuse of AI and noted that there are cases where notified bodies have told companies that their dossier isn't usable because they relied too heavily on AI and don't have confidence that the AI didn't make things up. Furthermore, he highlighted that as AI helps speed up certain functions, companies often expect workers to work faster and produce more output, which he said may ultimately be detrimental to quality.
"You have to define everything upfront," said Bos. "Start before you do anything by thinking about your strategy and keeping that control."
Wright echoed similar sentiment.
"It's about intention and being very intentional about how you deploy AI tools, both on the notified body side and on the manufacturer side," said Wright. "Whether we're talking about the need for human in the loop, the first point is to do that decomposition task.
"We talked about that from the notified body perspective, but I think manufacturers can do it too," he added. "If that approach is taken, then the quality of the submission can be excellent."
Fischer said it is important to build safeguards to prevent users from becoming complacent. He said that an AI system may function in a certain way a hundred times over, only to eventually produce a completely different output. So far, he said he hasn't seen a completely convincing safeguard system to prevent that. However, he said that AI safeguard developers may learn from cybersecurity processes by continuously testing a system, in which users receive test emails they must report to the cybersecurity department.
"One could consider similar ideas for building in mistakes, keeping people on their toes, and when using AI assistance to avoid that automation bias," said Fischer. "It should be risk-based and depend on what the AI is doing, but if it's really going down into doing TD assessments, informing decisions, then you have to consider these things."
He noted that there's also a lot of hype around AI, and regulatory professionals should ask whether they need to use AI for any specific task.
Bos said that, in general, regulatory professionals need to remain relevant and trustworthy when considering the use of AI tools.
"Whatever tools you use, make sure you are in command, in control of what you do," he said. "But also, be cautious when you work together with your colleagues so that they have the same behavior.
"The human in the loop means to me that the ethical, conscious judgment of the person should be in the loop, should be in control," he added. "We matter more than ever before."
RAPS AI in Regulatory Conference