The US Food and Drug Administration (FDA) on Thursday released draft guidance for industry on the problems posed by multiple endpoints in the analysis and interpretation of study results and how these problems can be managed in drug and biologic clinical trials.
The 54-page draft guidance looks to provide greater detail than ICH’s E9 Statistical Principles for Clinical Trials and clarify when and how multiplicity due to multiple endpoints should be managed to avoid reaching false conclusions.
“As the number of endpoints analyzed in a single trial increases, the likelihood of making false conclusions about a drug’s effects with respect to one or more of those endpoints becomes a concern if there is not appropriate adjustment for multiplicity,” FDA says. “The purpose of this guidance is to describe various strategies for grouping and ordering endpoints for analysis and applying some well-recognized statistical methods for managing multiplicity within a study in order to control the chance of making erroneous conclusions about a drug’s effects.”
The guidance goes on to describe the three families of endpoints: primary, secondary and exploratory, noting that when there “is more than one primary endpoint and success on any one alone could be considered sufficient to demonstrate the drug’s effectiveness, the rate of falsely concluding the drug is effective is increased due to multiple comparisons.”
Also described is the statistical analysis associated with a hypothesis test, which produces three primary measures of interest: a point estimate, a confidence interval and a p-value.
“In a clinical trial with a single endpoint tested at α = 0.05, the probability of finding a difference between the treatment group and a control group by chance alone is at most 0.05 (a 5 percent chance),” FDA explains. “By contrast, if there are two independent endpoints, each tested at α = 0.05, and if success on either endpoint by itself would lead to a conclusion of a drug effect, there is a multiplicity problem. For each endpoint individually, there is at most a 5 percent chance of finding a treatment effect when there is no effect on the endpoint, and the chance of erroneously finding a treatment effect on at least one of the endpoints (a false positive finding) is about 10 percent.”
More specifically, FDA says in the guidance’s conclusion that the chance of concluding that a drug is beneficial when it is not “is of primary concern” to the agency.
“The widely accepted standard is to control the chance of coming to a false positive conclusion (Type I error probability) about a drug’s effects to less than 2.5 percent (1 in 40 chance),” the guidance adds. “As the number of endpoints or analyses increases, the probability of making a false positive conclusion can increase well beyond the 2.5 percent standard. Multiplicity adjustments, as described in this guidance, provide a means for controlling Type I error when there are multiple analyses of the drug’s effects. There are many strategies and/or choices of methods that may be used, as appropriate, as described in this guidance”
The guidance also addresses times when companies tried to go back and find a positive result from a failed study.
“In the past, it was not uncommon, after the study was unblinded and analyzed, to see a variety of post hoc adjustments of design features (e.g., endpoints, analyses), usually plausible on their face, to attempt to elicit a positive study result from a failed study — a practice sometimes referred to as data-dredging,” FDA explains. “Although post hoc analyses of trials that fail on their prospectively specified endpoints may be useful for generating hypotheses for future testing, they do not yield definitive results. The results of such analyses can be biased because the choice of analyses can be influenced by a desire for success.”
The guidance also discusses the Bonferroni method, the Holm procedure, the Hochberg procedure, the Prospective Alpha Allocation Scheme, the fixed-sequence method, the fallback method, gatekeeping testing strategies and the truncated Holm and Hochberg Procedures for Parallel Gatekeeping, among others.
Multiple Endpoints in Clinical Trials: Draft Guidance for Industry