Best Practices for Statistically Validating Modeling and Simulation (MandS) Tools Used in Operational Testing

• Activity Number: 370
• Type: Contributed
• Date/Time: Tuesday, August 11, 2015 : 10:30 AM to 12:20 PM
• Sponsor: Section on Statistics in Defense and National Security
• Author(s): Kelly McGinnity* and Rebecca Dickinson and Laura Freeman
• Companies: Institute for Defense Analyses and Institute for Defense Analyses and IDA
• Keywords: Modeling and Simulation ; Validation ; Experimental Design ; Operational Testing

Abstract

In many situations, collecting sufficient data to evaluate system performance against operationally realistic threats is not possible due to cost and resource restrictions, safety concerns, or lack of adequate or representative threats. Modeling and simulation tools that have been verified, validated, and accredited can be used instead of, or in addition to, live testing in order to facilitate a more complete evaluation of performance. Two key questions that frequently arise when planning an operational test are 1) if live trials are possible, which (and how many) points within the operational space should be chosen for optimal ability to verify and validate the M&S? and 2) once that data is collected, what is the best way to compare the live trials to the simulated trials for the purpose of validating the M&S? This talk will discuss various strategies for addressing these two questions. The best methodologies for designing and analyzing will vary depending on the goal of operational test and the type of model used in the simulation.