FDA Says Real-World Evidence Could Generate 'Incorrect or Unreliable Conclusions’

Regulatory NewsRegulatory News | 08 December 2016 |  By 

Fifteen top US Food and Drug Administration (FDA) officials published an article in the New England Journal of Medicine on Thursday calling into question some of the potential uses of real-world evidence (RWE) but also acknowledging that real world research and the concepts of a planned intervention and randomization “are entirely compatible."

Defining RWE

Rachel Sherman, deputy commissioner for medical products and tobacco in the Office of the Commissioner at FDA, writing alongside FDA Commissioner Robert Califf and Janet Woodcock, director of the Center for Drug Evaluation and Research, begin their piece by taking a step back and calling the meaning of the term RWE “elusive.”

Part of that elusiveness comes from the fact that the term encompasses many different forms of information derived from sources generally outside of randomized, controlled clinical trials and can include electronic health record (EHR) data, claims and billing data, product and disease registries and data from personal devices or health applications.

And while acknowledging that such evidence “can inform therapeutic development, outcomes research, patient care, research on health care systems, quality improvement, safety surveillance, and well-controlled effectiveness studies,” the authors caution that “the confluence of large data sets of uncertain quality and provenance, the facile analytic tools that can be used by nonexperts, and a shortage of researchers with adequate methodologic savvy could result in poorly conceived study and analytic designs that generate incorrect or unreliable conclusions.

“Accordingly, if we are to realize the full promise of such evidence, we must be clear about what it is and how it can be used most effectively, and we must have appropriate expectations about what it can tell us,” they write.

Having said that, the FDA authors, who say they will “address key concerns and publish draft guidance on how such evidence can be used to assess safety and effectiveness in both premarketing and postmarketing regulatory requirements,” also conclude that RWE and the concepts of a planned intervention and randomization “are entirely compatible.

“Indeed, one of the most important advances in clinical trial methodology may be the broadening of the application of randomization outside more typical venues for clinical trials, such as academic research centers. But in order to gain collective confidence in the appropriate uses of this array of methods across disparate settings, we must first be clear about our terminology and its application,” they write.

21st Century Cures Act, PDUFA VI and RWE

The article’s release is timely, as the agency will be tasked (thanks to the 21st Century Cures Act, which passed the Senate on Wednesday and is expected be signed by President Barack Obama) with establishing a program within two years of the bill’s enactment to evaluate the potential use of RWE to help support the approval of a new indication for a drug or to help support or satisfy postapproval requirements.

The legislation defines RWE as “data regarding the usage, or the potential benefits or risks, of a drug derived from sources other than randomized clinical trials,” including sources such as “ongoing safety surveillance, observational studies, registries, claims, and patient-centered outcomes research activities.”

And not later than five years after the bill’s enactment, FDA would also have to issue draft guidance to industry on the use of RWE.

Similarly, under the reauthorized Prescription Drug User Fee Act (PDUFA VI), which is due for completion next year by Congress, FDA says it will conduct a public workshop and launch a pilot project or methodology development projects to identify how RWE can contribute to regulatory decisions.

In perhaps a sign of what’s to come in those future plans, the officials write that EHRs, registries and claims databases contain rich data already gathered in real-world settings at the point of care, and personal devices and apps can allow continuous monitoring and data capture and facilitate shared decision making. 


“But these data sources also raise concerns. EHR and claims data are not collected or organized with the goal of supporting research, nor have they typically been optimized for such purposes, and the accuracy and reliability of data gathered by many personal devices and health-related apps are unknown,” the authors write. “Furthermore, the use of any of these sources, including social media, raises important questions about the quality of the data they provide and about privacy.”

Limitations to the use of RWE “are particularly problematic when an observational study is used to evaluate the effectiveness of a medical product and the expected or observed effect is relatively small,” they continue. “When this is the case, it can be difficult to be confident that the effect is not due largely or wholly to confounding factors. This problem, compounded by the fact that observational studies often leverage existing rather than prospectively collected data (e.g., as part of a disease or product registry with well-established quality standards), can add to the uncertainty regarding findings and limit the usefulness of such data.”



© 2023 Regulatory Affairs Professionals Society.

Discover more of what matters to you

No taxonomy