Case Study Demonstration of the Application of Experimental Design to Operational Testing

• Activity Number: 179
• Type: Contributed
• Date/Time: Monday, August 4, 2014 : 10:30 AM to 12:20 PM
• Sponsor: Section on Statistics in Defense and National Security
• Author(s): Kelly McGinnity*+ and Laura Freeman
• Companies: Institute for Defense Analyses and IDA
• Keywords: Operational Testing ; Experimental Design

Abstract

Operational testing and evaluation is an essential part of the U.S. DoD's acquisition process. Via a case study on a mine neutralization system, we demonstrate the application of experimental design and statistical analysis to operational testing, discuss challenges, and propose potential solutions. The ability to systematically vary certain factors of interest is inherently limited by environmental qualities of the location of the mine field used during testing. Sequential testing (informing a future test design based on previous phases of testing and combining the results) is used address this challenge. One of the main response variables of interest for determining the effectiveness of a mine countermeasures system is the time it takes to detect and neutralize a mine. If the system does not perform this task within a certain amount of time, the event of interest is not observed, and the data point becomes censored. A Monte Carlo approach to calculating power for a designed experiment with censored data is proposed, and a statistical survival modeling approach to analyzing the results of the experiment is discussed.