Experts: Usability testing integral to evaluating med devices and IVDs

BRUSSELS – Device usability is a key component of the evaluation process and should neither be forgotten nor left to the last minute, experts told attendees at the 2023 RAPS European Clinical and Risk Management Conference.

Under the EU Medical Device Regulation (MDR) and the In Vitro Diagnostic Medical Devices Regulation (IVDR), usability testing needs to be performed and could be critical to determining whether a product will get or keep its certification, experts said.

“Usability testing should be performed in connection with risk management activities during design and development,” said Isabel Koch, team lead for IVDR at TÜV SÜD Product Service in Germany, who provided an overview of the regulatory requirements on usability testing for individual diagnostic self-tests and near-patient tests from the notified body perspective.

Usability and risk management reporting should be aligned and could theoretically be combined, but are usually written up separately, added Léon Assink, a senior consultant for the independent consultancy company Medical Device Project B.V. in the Netherlands.

Importantly, usability testing shouldn’t be an afterthought, as Torsten Gruchmann, Managing Director and founder of the independent consulting company Use-Lab based in Steinfurt, Germany, pointed out. “It is so important to talk about usability engineering for medical products,” he said.

Medical mistakes, some of which come down to medical device deficits, are reportedly the third leading cause of death, with as many as 250,000 cases per year reported in the US alone in 2014, Gruchmann said. “I think this is sufficient reason to focus on usability engineering as one solution to improve patient safety, and to reduce the number of deaths,” he said.

For medical devices, design deficits may be due to a variety of causes, such as insufficient communication between the developer and the end user, and insufficient evaluation. “Very often we learn that the developers of a medical product have never been in a hospital, so they never talk to the actual users. They have never seen the user environment, so they don’t have this feedback, which is quite important,” Gruchmann said.

It is vital to consider how the device is going to be used both in terms of the user requirements and the environment, agreed Koch. That includes considering the different skill levels of the various clinical users, or patients in the case of self-tests.

While there is no dedicated chapter on the usability of self-tests and near-patients tests in the IVDR, Koch said, Annex I provides some information on the general safety and performance requirements, some of which relate to usability.

Testing should be done in the population and the setting intended for final use, Koch said. Additionally, it is also important to have adequate measures in place to ensure the correct usage, such as clear and unambiguous instructions for use.

There are several parallels between risk management and usability engineering, according to Assink. They both start with the use in mind, although the risk management plan is more concerned with how a product will be used than who will be using it, he said.

Gruchmann offered some general advice on how to perform usability tests. Whether usability testing is being performed for a medical device or IVD, the first step is to determine the goals. A draft test plan should then be developed alongside a screening document for potential participants. After recruiting participants, the next stage is to conduct a pilot. From the pilot, a final test plan can be finalized and conducted, and after data consolidation and analysis, the final test report can be prepared, and the appropriate briefing documentation undertaken, he said.

There are important differences, he noted, in usability tests being performed to inform and help to improve the device design, and those that are performed to show the final usability of the product.

Formative usability tests should follow an iterative process involving different user groups and “thinking aloud” or informal assessments. Summative evaluation, on the other hand, concerns the final assessment of the design with regard to residual risks. This is where traceability becomes important, and evaluations must be formal, Gruchmann said.

“All this information is fed back to the manufacturer with the final risk assessment, according to ISO4971, and they have to decide whether the remaining risk is acceptable,” Gruchmann said. If there is a need for design changes here, then the summative evaluation must be repeated.

Usability testing must be done early in the design process, Gruchmann emphasized. “The sooner you fail, the earlier you reach the goal,” he said.

Experts: Usability testing integral to evaluating med devices and IVDs

Related topics

Recommended content

Welcome to the new RAPS Digital Experience