When are RCTs Required for Breakthrough Drugs, HDEs? JAMA Study Investigates
A new study in JAMA Network Open finds that drugs and medical devices approved by the US Food and Drug Administration (FDA) based solely on non-randomized controlled trials (RCTs) exhibited larger effect sizes compared to products that required further study in randomized trials. But the authors called for greater FDA transparency to instill confidence that decisions about drug and device approvals based on observational data are appropriate.The goal of the study, conducted by researchers including Stanford Professor John Ioannidis, Cochrane Founder Lian Chalmers and City of Hope Hematologist Benjamin Djulbegovic, was to identify how large treatment effects must be for regulators to consider data from non-RCTs sufficient for approval, as RCTs are considered the “gold standard” for generating clinical evidence for medical interventions.
“The main concern about non-RCTs is that it is currently challenging to predict with confidence when the results of RCTs will be concordant with those of observational studies. The consequences of such a discordance can sometimes be disastrous,” the authors write, pointing to a large randomized study conducted by the Women’s Health Initiative (WHI) to determine whether estrogen plus progestin could prevent coronary heart disease (CHD) and other chronic conditions in postmenopausal women. Contrary to earlier observational evidence, there were more cases of CHD and breast cancer in the treatment arm of the WHI study.
In the JAMA study, the authors found that 10% of drugs and biologics approved with breakthrough therapy designation (BTD) and devices with humanitarian device exemptions (HDEs) were approved based on non-RCTs, with “the magnitude of treatment effects … approximately 2.5-fold higher among treatments that were approved based on non-RCTs than among treatments that required further testing in RCTs.”
However, the study failed to identify a clear threshold for treatment effect above which the FDA would not require additional RCTs and found that only 1-2% of drugs approved solely based on non-RCTs “displayed so-called dramatic effects.”
In a previous study, the authors found that the European Medicines Agency (EMA) “approved 7% of drugs based on nonrandomized studies alone,” with 2-4% exhibiting dramatic effects. That study also drew an association between larger effect sizes in nonrandomized studies and higher rates of approval.
“The current study confirms observations we made evaluating EMA approvals, indicating that when larger magnitudes of effects are seen in non-RCTs, there is less desire from the regulatory agency to require subsequent confirmatory RCTs,” the authors write.
In total the authors reviewed 606 applications for breakthrough designation for drugs and biologics from 2012-2017 and 71 applications for humanitarian device exemptions from 1996-2017. Of those, the authors identified 36 drugs and biologics with BTD and 32 devices with HDE that were later approved based on non-RCTs and met other criteria for inclusion in the study.
Within those 68 interventions, the authors found that 17 (25%) required additional study via RCTs, while 51 (75%) did not.
Among the products with BTD, the authors found that most of the studies were prospective (94%) and the approvals based on data from single study arms (89%). The historical comparators used for these studies were mostly active treatments (86%) though the remaining studies relied non-active comparators such as natural history.
Among the 32 devices studies, 25 (78%) were prospective, 6 (19%) were retrospective and 1 (3%) included both prospective and retrospective data. Most of the studies were single arm (94%), though comparators were less likely to be active than for drugs and biologics with BTD, with 56% of the studies using an active treatment as a comparator and 14 relying on natural history or a comparison with baseline values.
“We concluded that the FDA has not requested subsequent RCTs when a treatment has displayed a larger magnitude of estimated treatment effects. However, the quality of data on which the FDA and EMA presumptively make these estimates and eventually decisions for approval is very low,” the authors concluded.
JAMA