The Learning Portal will be under maintenance Saturday, 8 August between 6 AM and 6 PM EST. Portal functionality will be unavailable during this window.
We apologize for any inconvenience caused during this time.

RAPS is closely monitoring developments in the Coronavirus (COVID-19) outbreak. See our public safety page for the latest updates.

 
Regulatory Focus™ > News Articles > 2020 > 2 > Estimating the Probability of Regulatory Registration Success

Estimating the Probability of Regulatory Registration Success

Posted 13 February 2020 | By Jennifer R. Weidman, PhD and Kimberly Belsky, MS 

Estimating the Probability of Regulatory Registration Success

This article discusses how to estimate the probability of regulatory/registration success for pharmaceutical products in development. The authors define the factors and methods used to assess regulatory/registration success and offer objective and data-driven methods to define probabilities for the purposes of documenting specific factors and/or risks relative to a likelihood of success. They provide a case study demonstrating how to use data and make estimate adjustments based on a variety of factors specific to individual programs. While this example applies to a drug and the US Food and Drug Administration (FDA), the same strategic approach can be applied to biologics, medical devices and combination products under review with any global regulatory Health Authority (HA).
 
The article is based on a presentation by the authors at RAPS Regulatory Convergence in September 2019.
 
Introduction
 
Determining the Probability of Regulatory Success (PRS) for a product approval is a key driver for ensuring business/market readiness, including prioritization for the purposes of budgeting and resource planning. When considering PRS, a “decision sciences” approach focusing on facts is critical for avoiding biases. To do this consistently requires an ability to evaluate a program and create an objective PRS in a concise, well-documented, transparent manner along with objective, analytically driven decision-making. While a “gut feeling” about regulatory success may be based on experience and regulatory acumen, it cannot be objectively revised for changes in the regulatory environment and does not allow for a documented, well-vetted strategic and holistic analysis of the risks (downward adjustment) or strengths (upward adjustment) of regulatory considerations. Furthermore, “gut-feeling” does not support development of actionable mitigation strategies based on facts. Thus, this approach provides an objective framework for forming comparable assessments across a variety of projects.
 
Technical vs. Regulatory Success
 
It is important to note that technical and regulatory success are different objectives and need to be assessed separately. Probability of Technical Success (PTS) applies to the probability a given clinical trial/study will be successful based on pre-defined endpoints, feasibility and other factors. Probability of Regulatory Success (PRS) is about whether FDA or another regulatory authority will grant approval for a product and based on factors within the scope of regulatory affairs, which often includes evaluation of the HA’s perspective regarding the clinical relevance for a particular endpoint as it applies to suitability for defining efficacy in the context of an approval application for registration. The overall probability of success will need to consider both the PRS and the PTS and is calculated as PTRS = PRSxPTS.
 
Six Elements of Good Decision Making
 
There are six elements for good decision-making.
 
Figure 1. Six Elements1


 
First, the information used must be meaningful and reliable. Second, the values and “trade-offs” must be clearly stated. This is necessary to ensure logically correct reasoning in support of the values and trade-offs. Each element is a critical component to the process and will ultimately drive the development of appropriate risk and mitigation plans.
 
Adhering to these six principles requires a method for estimating likelihoods of both technical and regulatory success. The method should be based on a common benchmark as a starting point, and be consistent across all programs and, perhaps most importantly, be well-documented. It is also important to have estimates facilitated by a neutral, non-affiliated facilitator with assessment sessions held with senior-level experts to help assess probabilities.
 
Looking at some starting benchmarks is useful as there have been several public domain studies which, on the average, have attained a 90% success rate for small molecule product registration success from Phase 3 onward.
 
Figure 2. Small Molecule Benchmark Results*


Each phase of development assumes each preceding phase was successful, if the program is an “average” program, leading to a 90% success rate. It is important to have discussions regarding reasons to believe, or not believe, regulatory success will be an endpoint. Upward adjustments, such as having a breakthrough or Regenerative Medicine Advanced Therapy (RMAT) designations or a Special Protocol Assessment (SPA), can improve a development program’s potential for success due to the increased engagement with FDA. Downward adjustments, such as not having adhered to FDA feedback, can lower the potential for regulatory/registration success.
 
Other upward adjustments include:
 
  • upward adjustment for a product with special designation (such as those in development for “ultra-rare” indications)
  • medicinal products to address unmet medical need in the treatment of a serious or life-threatening condition
  • the need for an advisory committee (US)
  • compliance with health authority advice
  • compliance with health authority data requests/analyses
  • robust clinical data
  • the targeted condition is on a health authority list of critical therapies
  • secondary indications, such as a more robustly sound safety profile when compared to a predicate product
  • the need for an advisory committee (US)
 
Potential downward adjustments include:
 
  • safety signals cannot be mitigated
  • therapeutic area science is evolving or unclear/not known
  • efficacy is not comparable to current stand of care
  • a magnitude of benefit is not expected
  • a change in the competitive landscape impacting comparator used
  • the need for an advisory committee (US)
  • compliance/inspection issues
  • non-compliance with HA advice
  • unmitigated technical risk
  • other approvability risks
 
The PRS Assessment Sheet
 
Figure 3. PRS Assessment Example


 
The PRS Assessment worksheet will list and juxtapose all applying upward and downward adjustments listed above. Identifying all upward and downward adjustments is the first step to the process. Each factor may not assigned a standard value (e.g., while an Advisory Committee can be a negative for a product with a serious and unmitigated safety considerations, it also can be a positive to highlight the product’s robust safety profile). And, while some may assign values to each such as -10% or +10%, there is not a 1-for-1 weighting as the weight of these factors can vary based on the drug, the therapeutic area, being first-in-class, etc.) The worksheet includes caveats and considerations in the “assessment notes” to help provide relevant context to factors considered. PRS includes successful interactions with regulatory bodies; PRS does not, however, address commercial viability or commercial success. For example, PRS does not consider generic competition entry into the market or off-label use of a competitor product by a healthcare professional. Technical probabilities (such as the likelihood a clinical study will render positive outcomes for its clinical endpoints) are addressed via the PTS. Probabilities are the quantitative values – the output – of this process.
 
A Case Study
 
Modified Alkaline Phosphatase (MAP) is an intestinal human alkaline phosphatase sequence modified during manufacturing to render an amino acid macromolecule. At present, MAP is in development as a treatment for sepsis associated with acute kidney injury. The Investigational New Drug (IND) application for MAP was filed with the Division of Cardiovascular and Renal Products, FDA’s Center for Drug Evaluation and Research (CDER) and the program received orphan designation for hypophosphatasia as well as fast track designation.
 
The regulatory pathway for a US registration of MAP based on a single Phase 3 study demonstrating significance for a primary endpoint predefined as overall survival appears to be straightforward. However, Phase 2 clinical study with MAP failed to meet its primary endpoints. The sponsor conducted a post-hoc analysis that trends toward a mortality benefit for patients with MAP. Despite a lack of evidence supporting a plausible mechanism by which treatment with MAP could mediate an effect on mortality, the agency has provided positive suggestions on how to enrich the Phase 3 study design. Feedback indicates FDA is open to a single study sufficiently able to support filing from both an efficacy and safety standpoint. The encouraging feedback is:
 
  • The current preclinical program is suitable to support clinical dosing up to three days in duration, as confirmed during a Type C meeting held prior to Phase 2 development and should be adequate to support a marketing application under these dosing parameters.
  • Phase 2 investigation supports adequate tolerability and safety in patients, yet the study failed to meet its efficacy endpoints. The sponsor concludes this failure is attributable to complications in trail design and not reflective of a lack of efficacy for MAP.
  • The sponsor conducted a post-hoc analyses of the Phase 2 trial data and identified an associated conclusion of mortality benefit for those patients treated with MAP. The agency is clearly skeptical of this conclusion, as evidenced by the feedback rendered with rejection of breakthrough therapy designation.
  • A postapproval pediatric study will be required, per the Pediatric Research Equity Act (PREA).
  • Thorough review of all historical correspondence between the sponsor and FDA indicates no additional postapproval study requirements have been identified.
 
Probability of Regulatory Success (PRS) for Approval of Application for MAT to Treat Patients With Sepsis Associated With Acute Kidney Injury Based on a Single, Phase 3 Dataset
 
Upward adjustments include:
 
  • Indication under investigation is characterized by significant unmet need in a clinical condition leading to serious outcome (e.g., death).
  • Agreement regarding the adequacy of the preclinical dossier to support marketing registration was sought and achieved via Type C meeting.
  • Have received comprehensive FDA feedback on the Phase 3 clinical development program to support filing of marketing application based on this single Phase 3 study.
  • FDA agreement on the primary endpoint
  • Agreement on single study to support application (“Assuming the study is conducted well and MAP is shown to reduce the risk of mortality, the available data are likely adequate to support a marketing application.”)
  • Agreement on size of planned safety database at filing
 
Downward adjustments include:
 
  • No Special Protocol Assessment (SPA) for this evolving therapeutic area.
  • Related, competitor’s product received a Complete Response Letter (CRL) due to lack of significant clinical benefit demonstrated in relation to safety profile.
  • Potential for an Advisory Committee meeting due to first-in-class status in an emerging therapeutic area.
  • Together with Phase 2 findings upon treatment of patients with MAP, recent regulatory decision on competitor’s product suggests one key discussion point will be the overall benefit risk for MAP in this patient population.
  • Manufacturing site contracted to manufacture clinical trial material recently received a GMP Warning Letter.
  • Worst case: all clinical data collected to date derived from this manufacturing site is invalidated due to erroneous record-keeping or actions on behalf of the Chief Medical Officer (CMO).
 
For this case study, the “PRS Estimate” for the US is 65%. The completed PRS worksheet would include the upward and downward adjustments noted above, the “PRS Rationale” would include the following:  
Key to this PRS are the assumptions the Phase 3 study design will be validated by SPA-agreement and that compliance issues at the CMO level have no impact on the quality of Clinical Trial Material (CTM).
 
Mitigation Strategies
 
Once the PRS is determined with assumptions noted as appropriate, mitigation strategies can be developed. One might propose actions such as the following for this case study:
 
  • Discussion with FDA regarding the design and endpoints of the single Phase 3 study intended to support an application for registration has yielded clear and thoughtful feedback on the design elements proposed by the sponsor as well as additional key points aimed at designing a clinical trial suitable to support a marketing application.
  • Altogether, with incorporation of FDA suggestions on the Phase 3 study design to date, a SPA should be requested upon finalization of the protocol and (at minimum) be provided with a proposal for a robust statistical analysis plan (SAP). This should effectively mitigate any outstanding risk associated with initiating a Phase 3 study, falling short of agency expectations of suitability based on design alone; upon securing SPA agreement, the PRS will be re-assessed.
  • Investigation of compliance issues at manufacturing site is on-going; quality will conduct a compliance risk-assessment; PRS will be adjusted accordingly.
 
A Look at History
 
Data from five benchmarking studies have shown, with the only significant agreement between studies, how large molecules have had a higher probability of success than small molecules. Large molecule Phase 1 registration success rate was 21%. Comparatively, small molecule Phase 1 registration success rate was 10%. Rare disease subsets have widely varying benchmarks.2-6
 
Figure 4. Small and Large Molecule History


 
Conclusion
 
Estimating the probability of success for regulatory approval can be carried out in a consistent, objective manner using a concise, transparent and well documented process. Upward and downward factors and assumptions made can be revisited and adjusted as new information arises. This objectively and analytically driven decision-making process provides greater benefits than can be found through “gut feelings” or other non-objective methods. Furthermore, leveraging “decision science” and focusing on facts allows for the development of mitigation strategies for reducing risks associated with regulatory/registration success. The “take away” message is one showing estimating regulatory/registration success probability can be a reality and is critical for managing expectations throughout the organization.
 
References
 
  1. Image courtesy of Executive Briefing. 2007 Strategic Decisions Group. www.sdg.com. https://image.slidesharecdn.com/decisionleadershipsdgstanforduniversity-13081693685566-phpapp01-110615152438-phpapp01/95/decision-leadership-sdg-stanford-university-7-728.jpg?cb=1308151509. Accessed 11 February 2020.
  2. Hay, et al, “Clinical Development Success Rates for Investigational Drugs.” Nat Biotechnol. 2014. Jan:32(1);40-51. https://www.ncbi.nlm.nih.gov/pubmed/24406927. Accessed 10 February 2020.
  3. “Clinical Development Success Rates 2006-2015.” Biomedtracker. 2016.
  4. DiMasi, et al. “Trends in Risks Associated with New Drug Development: Success Rates for Investigational Drugs.” Clinical Pharmacology and Therapeutics. 87(3):272-7. March 2010. https://www.researchgate.net/publication/41405454_Trends_in_Risks_Associated_With_New_Drug_Development_Success_Rates_for_Investigational_Drugs. Accessed 10 February 2020.
  5. Smietana, et al. “Trends in Clinical Success Rates.” Nat Rev Drug Discov. 2016. Jun:15(6);379-80. https://www.ncbi.nlm.nih.gov/pubmed/27199245. Accessed 10 February 2020.
  6. Wong, et al. “Estimation of Clinical Trial Success Rates and Related Parameters.” Biostatistics. Volume 20, Issue 2, April 2019, Page 366. https://academic.oup.com/biostatistics/article/20/2/366/5183543. Accessed 10 February 2020.
 
Further Reading
 
  • Kahneman D. “Thinking Fast and Slow.” Farrar, Straus and Giroux. October 2011.
  • Speztzler C. “Decision Quality Value Creation from Better Business Decisions.” Strategic Decisions Group International LLC. 2016.
  • Taleb N. “Fooled by Randomness: The Hidden Role of Chance in Life and in the Markets.” Random House. 2005.
 
About the Authors
 
Kimberly Belsky, MS, is executive director, regulatory policy and intelligence, Mallinckrodt Pharmaceuticals. She may be contacted at Kimberly.belsky@mnk.com.
 
Jennifer R. Weidman, PhD, molecular genetics, is senior director, regulatory affairs, Mallinckrodt Pharmaceuticals. She may be contacted at Jennifer.weidman@mnk.com.
 
Acknowledgement: the authors would like to recognize Rodger Thompson, president, Sierra Decision Consulting, who was an integral part of the authors’ presentation at RAPS Regulatory Convergence in September 2019.
 
Cite as: Weidman J R and K Belsky “Estimating the Probability of Regulatory Registration Success.” Regulatory Focus. February 2020. Regulatory Affairs Professionals Society.
 

Regulatory Focus newsletters

All the biggest regulatory news and happenings.

Subscribe