Modeling and simulation methods have gained increased acceptance and are becoming more common in the drug and medical product development process. Modeling and simulation hold the promise of increasing the efficiency of drug discovery by identifying suitable products for development, while potentially reducing overall costs. Many modern statistical methods including resampling, bootstrapping, Bayesian statistics and Monte Carlo methods depend almost entirely on simulations.

The first part of this talk highlights a general approach to establishing credible models and simulations, and is based on a working group of the Scientific Computing Board at the US FDA. In this section we outline twelve steps for good modeling and simulation practices.

In the second part of this talk, we discuss a common deficiency with many statistical simulations, namely that implicit claims of accuracy and precision are much stronger than can be justified. Focusing on binomial proportion estimation, we use R to establish that frequently employed simulation sizes n=1,000 or n=10,000 are generally inadequate for commonly used levels of precision. Using both standard normal approximations and exact methods, we establish the required number of replications can approach 4,000,000 for some scenarios of interest. Additionally, we show that simple quantile estimation using simulation, a method used with Bayesian estimation, as well as confidence interval estimation in conjunction with naive resampling and bootstrap methods, is fraught with potential problems. Finally, we suggest possible methods to enable large scale simulation efforts, with an emphasis on parallel computing methods.