Empirical Signal-to-Noise Ratios from Operational Test Data

• Activity Number: 179
• Type: Contributed
• Date/Time: Monday, August 4, 2014 : 10:30 AM to 12:20 PM
• Sponsor: Section on Statistics in Defense and National Security
• Author(s): Matthew Avery*+
• Companies: Institute for Defense Analyses
• Keywords: Experimental Design ; Signal-to-Noise Ratio ; Defense

Abstract

Statistical power is a common metric for assessing experimental designs. While this metric depends on many factors, one of the most critical is the expect effect size of relevant factors and the relative noise expected in the data. Together, these values are summarized as the signal-to-noise ratio (SNR). Software packages like JMP 10 and Design Expert use SNR as a critical component in power calculations, and by general "rule of thumb", values such as 0.5, 1, and 2 are used. However, it is not clear that these values represent the true spectrum of likely outcomes from operational test data. Operational testing is the final phase prior to fielding in the DOD acquisition process for new systems. Due to the operational realism strived for in such testing, there are often many sources of uncontrolled variation, making it difficult to plan an appropriate test based on the SNR. In this talk, we summarize observed SNRs from a wide spectrum of operational tests and offer suggestions for the use of SNR in operational test design.