Paper Looks to Establish Best Practices for Evaluating Effectiveness of Regulations

A white paper released by Mini-Sentinel, and adverse event tracking pilot project, evaluates a series of existing studies evaluating the impact of the US Food and Drug Administration's regulatory actions, finding those studies to be largely lacking.

The paper, published 12 March and released to the public on 5 April, sought to answer two questions: what regulatory methods have been used by FDA, and what are the best practices for evaluating regulatory actions?

The two-phase study made five conclusions:

• rigorous evaluations of the impact of FDA regulatory actions have been infrequent, especially relative to the large number of actions implemented by the FDA. 
• when FDA regulatory actions were evaluated minimal research design standards were often not employed.  For instance, less than a quarter used a control group.   Assessments of data completeness or quality were usually not well described but were at least mentioned in most (86%) of the papers.
• many of the assessments were limited in scope and examined only changes in the use of the targeted medical product.  Studies that included a broader array of measures such as unintended impacts (e.g., increases in the use of substitute products and services) were uncommon.  Studies that included outcomes measures relating directly to patient health and adverse events were even rarer.  As a result, many evaluations of FDA regulatory actions used suboptimal research designs and analytic methods, making the results limited and susceptible to biased findings.  Overall, this review revealed considerable gaps in the evidence-base making any assessment of the impact of FDA policies premature at this time.
• there are novel methods, especially quasi-experimental research designs, which may be useful for evaluating the impact of FDA regulatory actions.  The methods identified by this panel can address many of the limitations of previously published evaluations. 
• the careful and consistent application of these novel methods can provide a valuable opportunity to achieve the goal of less biased assessments of FDA regulatory actions.

The study recommends that researchers in the future "use research designs and analytic methods that produce strong internal validity" with better control groups and pre-test measures to best evaluate FDA's regulatory actions.

Read more: