QN Allocation: Balancing the Number of Replicates vs. the Number of Treatments in a Designed Simulation Experiment

• Activity Number: 78
• Type: Contributed
• Date/Time: Sunday, August 4, 2013 : 4:00 PM to 5:50 PM
• Sponsor: Section on Physical and Engineering Sciences
• Author(s): Allan Mense*+ and Terril N. Hurst and Jarom Ballantyne
• Companies: Raytheon Company and Raytheon Missile Systems and Raytheon Missile Systems
• Keywords: Design of Experiments, DOE, Simulation

Abstract

APPLYING statistical design & analysis to simulation experiments has given rise to the following challenge, called QN allocation: Given a fixed number of runs M = Q x N, where N is the number of treatments and Q is the number of Monte Carlo draws per treatment (replicates), determine the right mix of treatments vs. replicates to answer confidently a specific question. An example question is, "Can we reject the null hypothesis that the expected value of system performance across its entire operational space meets or exceeds the expected value stated in the specification?" Another example is to evaluate Probability of Non-Compliance (PNC), a one- or two-sided tolerance interval bounding probability of success for a specified proportion of admissible scenarios within the operational population. Analysts at Raytheon Missile Systems have solved the QN allocation problem. Solutions have ranged from {Q, N} = {1, M} to {50, M/50}, depending directly upon the form of the stated requirement. The paper presents specific examples, explaining rationale for each solution.