Not all evidence is created equal, and understanding what claims can be made based on what evidence is important to remaining in compliance with US federal law and avoiding Untitled or Warning Letters from the US Food and Drug Administration (FDA).

That's the advice given by Julie Tibbets, a partner at Alaston & Bird LLP, at a recent conference hosted by the Food, Drug and Law Institute (FDLI) in Washington, DC.

While noting the general "downward trend in letters" sent by FDA's Office of Prescription Drug Promotion (OPDP), Tibbets said a majority of all letters sent to companies reference data from flawed studies that she said were "unable to provide substantial evidence for a claim," either by overstating the efficacy of a product or making unsubstantiated claims regarding its superiority relative to another product or intervention.

Of 19 letters sent by OPDP since October 2012, 11 cited inadequate evidence behind claims, violations that came in second only to the 13 letters that cited inadequate presentation of risk information.

FDA regulates this area under its authority of 21 CFR 202.1(e)(6), which states that a product is misbranded if its advertising represents or suggests that a product is safer or has fewer side effects than "demonstrated by substantial evidence," or if the study used is "inadequate in design, scope or conduct" to furnish substantial evidence.

Substantial Evidence

It is this notion of "substantial evidence" that earns many companies an Untitled Letter, said Tibbets. She cited an April 2013 Untitled Letter to Teva in which OPDP took issue with claims it said were unsupported by evidence.

Regulators wrote that Teva's materials state or otherwise imply that Clozapine, an antipsychotic, is superior to other studied drugs like olanzapine, risperidone and haloperidol based on its "purported improvement in the excitement symptom cluster."

"These claims, however, are not supported by substantial evidence or substantial clinical experience," FDA wrote. That would require a head-to-head clinical trial to substantiate, it continued, which Teva had not conducted. Instead, the company had only looked at retrospective data to support its claim, which FDA deemed insufficient.

In general, the issue is that if a study is not designed to evaluate a given hypothesis, it will not be adequately powered or designed to test whether it is correct. For example, some companies will take data from a subgroup that performed well in a study, and claim that this shows the drug is especially effective in that population. However, in most cases the sample size is not adequate to account for error. In other cases, companies will compare data between two studies, claiming that its product outperforms a competitor. Without the use of a head-to-head trial, this may not be correct. Another example is grouping studies through the use of metanalyses. While this can prove useful for discovering adverse events, it is generally not an exact science given the variation between studies and the potential for publication and other biases.

These issues affected numerous companies, many of which have received Untitled Letters from OPDP and other FDA promotional review divisions.

The Eleven Types of Evidence You Shouldn't Reference

So what can companies do? Avoid using evidence obtained through certain habits, Tibbets said.

"These types of study designs were not able to offer substantial evidence in support for efficacy claims," Tibbets continued, referencing violations found in the eleven Untitled Letters issued since October 2012.

1. retrospective post hoc analyses
2. retrospective single- institution chart reviews
3. retrospective sub-group analyses
4. retrospective exploratory analyses
5. retrospective analyses of different study populations
6. post hoc analysis in a poster presentation
7. open-label study with no control group
8. comparative claims where not all competitors were studied
9. meta-analyses based on a literature review
10. lack of pre-specified endpoints for efficacy claims
11. patient diaries to support adherence claims

"In general, anything that starts with, 'retrospective' is something that should be a red flag for your review team," she concluded.

Tibbets' Presentation