Using Simulations for Regulatory Decision-Making: How Many Simulations Do We Need to Run?

• Activity Number: 372
• Type: Contributed
• Date/Time: Tuesday, August 11, 2015 : 10:30 AM to 12:20 PM
• Sponsor: Biopharmaceutical Section
• Author(s): Paul Schuette*
• Companies: FDA/CDER
• Keywords: Simulations ; Methodology ; Regulatory Science ; Sample Size

Abstract

Simulations are being used more frequently in statistical work and an increased use of simulations is called for in the FDA Regulatory Science Strategic Plan. However, many statistical simulations contain implicit claims of accuracy and precision are much stronger than can be justified. Focusing on binomial proportion estimation, we use R to establish that frequently employed simulation sizes N=1000 or N=10000 are generally inadequate for commonly used levels of precision. Using both standard normal approximations and exact methods, we establish the required number of replications can approach 4 million for some scenarios of interest. Additionally, we show how our results can be readily extended to other distributions such as Poisson distributions. Moreover, we show that simply increasing the number of simulations will not suffice if one is using a biased method, illustrating this point with simple quantile estimation for naive resampling and bootstrap methods. Finally, we suggest possible methods to enable large scale simulation efforts, with emphases on efficient implementation and parallel computing methods.