Experimental Design Challenges in Operational Testing: Recent Case Studies

• Activity Number: 545
• Type: Contributed
• Date/Time: Wednesday, August 7, 2013 : 10:30 AM to 12:20 PM
• Sponsor: Section on Statistics in Defense and National Security
• Author(s): Matthew Avery*+ and Laura June Freeman
• Companies: IDA and Institute for Defense Analyses
• Keywords: Experimental Design ; Case Study ; Defense ; National Security

Abstract

Operational testing is a crucial part of the Department of Defense's acquisition process. Statistically designed experiments provide a scientific rational for determining test adequacy and ensuring sufficient coverage across the operational range of the system. Goals of "spanning the operational envelope" and ensuring high power for detecting factor effects are constrained by matters of cost, resource availability (e.g., suitably trained operators for the new system, number of available units) and time. We discuss some examples of how these constraints inform experimental design decisions. First, we consider an aircraft-mounted radar jamming system. A two-tiered design based on DoE principles is employed to address the two primary test goals. This approach mitigates the risk associated with the inability to produce a high density threat environment in open air testing. The next example deals with the challenges of operational testing for an enterprise resource planning (ERP) system for the Navy. This design presents different challenges, including highly skewed data, density estimation, and nonparametric hypothesis testing.