How Many Simulations Do We Need to Run?

• Activity Number: 387
• Type: Roundtables
• Date/Time: Tuesday, August 11, 2015 : 12:30 PM to 1:50 PM
• Sponsor: Section on Statistical Computing
• Author(s): Paul Schuette*
• Companies: FDA/CDER
• Keywords: simulations ; computing ; sample size

Abstract

Simulations have become an important part of statistical computing and may be the only feasible approach for some scenarios. However, simulations also have limitations, which may not always be recognized. In particular, one can ask how many simulations are necessary to obtain a reasonable estimate for a value of interest. Many authors use N=1000 or N=10,000 as standard choices, yet it can be shown that such a choice yields only limited accuracy for some scenarios. For example, estimating a binomial proportion p with 10,000 simulations yields a result that is only reasonably accurate to at most two decimal places. It can be established that estimating a binomial proportion p to three decimal places can require nearly 4,000,000 simulations. Hence, authors may be overstating the accuracy of results obtained from simulations based on those of modest size, such as N=10,000. Fortunately, advances in computer technology such as 64-bit architecture, multicore, cluster, and cloud-based computing allow a large number of simulations to be readily and inexpensively generated.