Analysis: Real-world studies of cancer drugs suffer from poor quality
Real-world oncology data, which is increasingly being touted as a potential basis for regulatory and reimbursement decisions, tends to be of poor quality and may overstate the survival benefits of cancer drugs, according to a new analysis.The findings, which were published in the European Journal of Cancer, showed that while most cancer drugs approved in the United States and Europe had some type of real-world data (RWD) study available, the methodological quality was “poor” in about 80% of the studies evaluated. The study flaws included patient selection bias, lack of control of confounders and missing or insufficient survival data.
“While guidance on the design and conduct of RWD studies are available, the reality is that the term is used ubiquitously to cover all non-randomised studies that use routine health records. A major issue is that data from such studies can be used as evidence of effect in routinely managed populations without any explicit reference to their methodological quality,” Ajay Aggarwal, PhD, of the London School of Hygiene and Tropical Medicine, and colleagues wrote.
Despite increasing calls in the US and Europe to use RWD from electronic health records, claims databases, or disease registries in regulatory decision making, there has not been a systematic assessment of the quality of existing RWD for approved cancer drugs, according to the researchers. (RELATED: CDER looks ahead at real world evidence, Regulatory Focus 15 March 2019; FDA offers real-world evidence examples in device applications, Regulatory Focus 18 March 2021; ASCO-Friends meeting: NCI offers strategies to modernize cancer trials, Regulatory Focus 13 April 2021)
The retrospective cohort study included all published RWD studies that reported effectiveness of new cancer therapies approved by the US Food and Drug Administration (FDA) and the European Medicines Agency (EMA) between 1 January 2010 and 31 December 2015. The analysis was limited to antineoplastic and immunomodulating agents for solid tumors used in the non-curative setting. The researchers looked specifically at quality of the studies and to what extent survival outcomes were comparable between RWD and their pivotal randomized clinical trials.
Results highlight variation
From 2010 to 2015, the EMA and FDA approved 50 cancer medications across 57 solid tumor indications. The final study cohort included 293 RWD studies covering 45 of those 57 indications. Nearly all – 98% – of the RWD studies were cases series. Just six studies (2%) used data from national cancer registries, a trend the researchers described as worrying. “A major advantage of registry data relative to data from single or selected centres is the very large sample size and coverage of eligible patients, especially in single-payer systems,” they wrote.
When evaluating study quality using the Newcastle Ottawa Scale, the researchers found that nearly 80% or 230 studies were rated as “low” quality, with 22% of studies being rated as “moderate” quality, and no studies rating as “high” quality. When the researchers broke down study quality by funding source, they found that studies funded by the pharmaceutical industry – about 27% – scored better on quality than those without industry funding.
The researchers also analyzed survival differences between each drug’s pivotal clinical trial and the RWD reported. Overall, they found that 37% of studies had better survival outcomes than the pivotal trial while 63% of studies demonstrated worse survival. Lower quality RWD studies were more likely to show better survival outcomes over the pivotal study. The two findings suggest that the survival benefits in trials may not be carrying over into real-world patients, while at the same time low-quality studies have the potential to overstate the benefits of new cancer medications, according to the researchers.
Steps to improve quality
The researchers offered a couple of suggestions that could potentially help raise the quality of real-world evidence. They recommend that publishers require authors to complete a methodological checklist prior to study submission to ensure transparency or to have peer reviewers complete a critical appraisal of methods as part of their assessment. Additionally, the authors noted the importance of ensuring adequate funding to develop high-quality research programs.
“The standard of RWD studies of cancer drugs needs to improve and be more consistent prior to its routine use in support of clinical practice and policy change,” the researchers wrote.
The research was supported by grants from the National Institute for Health Research and the UK Research and Innovation Economic and Social Research Council. The researchers reported having no conflicts of interest.
European Journal of Cancer study
