Comparison of the Neyer D-Optimal and 3pod Sensitivity Test Designs

• Activity Number: 108
• Type: Invited
• Date/Time: Monday, August 4, 2014 : 8:30 AM to 10:20 AM
• Sponsor: Section on Statistics in Defense and National Security
• Author(s): Barry Neyer*+
• Companies: Excelitas Technologies
• Keywords: Sensitivity Test ; Threshold Test ; Optimal design ; SenTest

Abstract

The performances of the recently proposed 3pod sensitivity test design of Wu and Tian (2014), along with variations of the 3pod design, were compared to the original Neyer D-Optimality design (1994) used in the SenTest software through multiple simulated runs of different sizes, combinations of initial test parameters, and variations of test designs. Three performance metrics were used to evaluate the first two phases of the designs: percentage of tests without a zone of mixed responses ("wasted runs"), and the Mean Squared (truncated) Errors of both the mean and standard deviation. There was no overall design that had the best measure of performance for all metrics, sample size, or for all sets of test parameters. Designs that performed the best by one metric were not always the best when evaluated against other measures; designs that performed the best in some sets of test parameters performed worse in others. While there was little difference between various test designs for most sets of test parameters, there was a marked decrease in efficiency for the initial stage or sub phase of Wu and Tian when compared with the Neyer D-Optimal for test parameters substantially different from the underlying population. Additional simulation was conducted comparing various methods for extreme quantile estimation. The simulation showed that the c-optimal approach in the SenTest software yielded estimates with smaller MSE than the other designs studied.