The US Food and Drug Administration (FDA) on Tuesday released new draft guidance clarifying how it plans to determine cases when real-world data may be sufficient for use in premarket and postmarket regulatory decisions, without changing the standards used to make those decisions.

The 21-page draft document, which is open for comment for the next 90 days, also explains how the agency plans to evaluate real-world data to determine whether it may be sufficiently relevant and reliable for various regulatory decisions. It also clarifies when an Investigational Device Exemption (IDE) may be needed to prospectively collect and use real-world data for the purposes of determining the safety and effectiveness of a device.

Background

FDA makes clear that it believes real-world data, collected from sources outside of traditional clinical trials, can provide “powerful insight” into the benefits and risks of medical devices, including how they are used by health care providers and patients.  The draft guidance gives generalized examples taken from actual uses of real-world data as valid scientific evidence for various regulatory decisions.

In seeking to outline the characteristics and sources of real-world data that may be sufficient for use in making regulatory decisions, the draft guidance explains how the quality (i.e., relevance and reliability) of such data can vary greatly across sources.

The agency also points to verifiable source documentation, which it says is the origin of real-world data elements, and can include, but is not limited to: paper or electronic inpatient and outpatient medical records and case histories, diagnostic laboratory and imaging data, patient-reported outcome measures, and medical device performance data that exists within the device such as self-diagnostics, error codes and patient diagnoses/treatments delivered (including the unique device identifier (UDI)).

Likewise, there are many types of regulatory decisions with varying levels of evidentiary needs, and though FDA says its evidentiary standards for regulatory decisionmaking are not changing, the agency notes that it will evaluate whether available real-world data is of sufficient relevance and reliability to address a specific regulatory decision.

“This draft guidance is a cornerstone of our strategic priority of creating a national evaluation system for medical devices,” FDA says. “That system would build on and leverage the vast amount of data and information collected during the treatment and management of patients.”

And though FDA does not endorse one type of real-world data over another, it says that sources of that data should be selected “based on the ability to address specific regulatory questions. Collection of [real-world data] should not dictate, interfere with or alter the normal clinical care of the patient, including choice of treatment.”

In addition to real-world data, FDA also defines “real-world evidence,” which it says may potentially be used to understand device performance at different points in its life cycle, including but not limited to:

• generation of hypotheses to be tested in a prospective clinical study
• as a historical control, a prior in a Bayesian trial, or as one source of data in a hierarchical model or a hybrid data synthesis
• in a setting where a registry or some other systematic data collection mechanism exists and the data can potentially be used as a concurrent control group or as a mechanism for collecting data related to a clinical study to support device approval or clearance
• in some circumstances where real-world use of a device is in a broader patient population or wider set of circumstances than described in the device labeling, it may be possible to use existing systematically collected real-world data to expand the labeling
• for public health surveillance efforts and to understand the evolution of the benefits and risks of devices after they have been approved or cleared in the US

And as real-world data methodology and infrastructure grow, FDA says real-world evidence may be well-suited to address issues identified by FDA; to reduce the time and cost of evidence generation to meet postmarket requirements; or to be used in lieu of submitting individual Medical Device Reports (MDRs) or to provide postmarket data in lieu of some premarket data under the Expedited Access Pathway (EAP) program.

Relevance and Reliability

The relevance and reliability of the source of the real-world data and its specific elements are highlighted by FDA as important factors that FDA will assess.

“The underlying data should be robust (i.e., provide meaningful information under a variety of conditions) for the purposes and analyses for which it was designed. These assessments will be used to determine whether the data source(s) and the proposed analysis generate evidence that is sufficiently robust to be used for a given regulatory purpose. That is, the threshold for whether RWD is sufficiently relevant and reliable for use will depend on the level of quality required and/or necessary to make a particular regulatory decision,” FDA notes.

FDA will assess the reliability of the data and the data sources by evaluating several factors including: “how the data were collected (data accrual); whether the data as collected are complete, accurate and adequate for answering the question at hand (data adequacy); and whether the people and processes in place during data collection and analysis provide adequate assurance that bias is minimized and data quality and integrity are sufficient (data assurance).”
FDA will also assess the relevance of real-world data and the source as a part of the evaluation of the regulatory issue being addressed.

“Relevance of [real-world data] for regulatory decision-making can be assessed either prior to a regulatory submission such as via the pre-submission process, or during the regulatory review process.  Since data elements for existing RWD sources are determined in advance and are primarily chosen for non-regulatory purposes (e.g., quality assurance (QA) and quality improvement (QI) in the case of clinical care registries), FDA will assess whether the individual data elements contained within the existing RWD source are sufficient (i.e., complete, well-defined, and appropriate in scope and timing) to fulfill a regulatory purpose,” FDA says.

In terms of ensuring reliability, FDA says a “prospective protocol that pre-specifies the data elements to be collected, data element definitions (i.e., data dictionary to provide a common definitional framework), methods for data aggregation and documentation (e.g., common case report form, abstraction from verifiable sources), and the relevant time windows for data element utility and outcome assessments (i.e., common temporal framework) is essential.”

Key factors FDA will assess include:

• the preparedness of individual sites for complete and accurate collection of observational data (e.g., defined processes, site training and support, dedicated qualified personnel)
• use of a common data capture form; use of a common definitional framework (i.e., data dictionary)
• adherence to a common temporal framework for collection of key data points
• the data collection procedures, data evaluation protocol or statistical analysis plan including when the data collection procedures were developed relative to actual data evaluation (i.e., prospective vs. retrospective)
• the sources and technical methods used for data element capture (e.g., chart abstraction, point of care entry, EHR integration, UDI capture, data records from device, linkage to claims data);
• patient selection and enrollment criteria that minimize bias and ensure a representative real-world population (e.g., all-comer’s design, consecutive patient enrollment
• the timeliness of data entry, transmission, and availability
• whether the act of collection of data impacts the ability to measure treatment outcomes;
• whether necessary and adequate patient protections were in place (e.g., de-identified data, maintenance of privacy, and need for informed consent as determined by the reviewing IRB and in compliance with FDA regulations)

Examples

FDA also offered the following examples from actual regulatory uses of real world evidence for regulatory decision making:

• Expanded indications for use: FDA offers the example of an unidentified Class III device with narrowly defined indications that has seen an expansion of uses outside of approved indications. To address the lack of data to support the new indications, FDA says relevant medical societies have established a national registry to collect safety and effectiveness information for all patients implanted with this device and a study using the registry data collection and analysis infrastructure was initiated with an approved investigational device exemption (IDE) application since the study focused on a use of this device that was not within the approved indications for use and imposed collection of specific follow-up data that might not otherwise be performed as part of standard medical care. FDA says it “is hopeful that the data may address critical safety questions related to the use of these devices and may be of sufficient quality to help support labeling changes or other regulatory decisions.”
• Postmarket Surveillance Studies: FDA says it has issued a series of postmarket surveillance study orders, related to investigating patient safety issues in a type of class II device, under the authority of Section 522 of the Federal Food, Drug, and Cosmetic Act. The orders covered multiple devices from different manufacturers that are similar in intended use, design, and other characteristics. To comply with the orders, FDA says many manufacturers decided to collaborate with a clinical professional society in this field and with FDA to develop a patient 579 registry that would collect needed data to address the public health questions.
• Post-Approval Device Surveillance as Condition of Approval (ie. For permanent implants): “FDA has worked with manufacturers and professional societies to evaluate the registries and has found that they can be reliable for certain health outcomes of interest. Should a registry exist that is capable of addressing the questions for which a Post-Approval Study (PAS) may be issued, FDA instead may issue a condition of approval that a manufacturer participate in and support collection/reporting of registry data on their device in lieu of a condition of approval specifying a formal PAS.”
• Control Group: “A manufacturer approached FDA during the development of a new medical device that had substantial technological changes from previous iterations of that specific device and other similar devices from other manufacturers. FDA determined that additional clinical evidence was needed to support an approval decision for this device. A registry exists that captures all uses o medical devices in this clinical indication. The manufacturer designed a clinical study that compared the use of the new device to a non-randomized concurrent control group derived from the registry. The existing registry was evaluated by FDA and the manufacturer according to the factors cited in this guidance and was found to provide sufficient data on the control population.”
• Supplementary Data, which FDA says it has used in the past, in combination with case reports, publications, adverse event reports, engineering and nonclinical test data, to provide a full understanding of the severity of an issue, precipitating factors, affected population and alternative therapies.
• Objective Performance Criteria and Performance Goals, which refer to a numerical target value derived from historical data from clinical studies and/or registries and may be used in a dichotomous (pass/fail) manner by FDA for the review and comparison of safety or effectiveness endpoints.

FDA is accepting comments on the draft guidance for the next 90 days.

Use of Real-World Evidence to Support Regulatory Decision-Making for Medical Devices: FDA Draft Guidance for Industry and Staff

Federal Register