Scaling System Test Software Failure Rate Estimates For Use In Field

• Activity Number: 150
• Type: Contributed
• Date/Time: Monday, August 12, 2002 : 2:00 PM to 3:50 PM
• Sponsor: Section on Physical & Engineering Sciences*
• Author(s): Xuemei Zhang*+ and Daniel Jeske+ and Loan Pham
• Affiliation(s): Bell Labs, Lucent Technologies and Bell Labs, Lucent Technologies and United Technologies
• Keywords: Software Reliability Growth Models ; Likelihood Ratio Test ; Operational Profile Testing ; Software Failure Rate

Abstract

An estimate of the user-perceived failure rate of software is essential for deciding when to deploy the software while maintaining an appropriate balance between reliability and time-to-market. Typically, software reliability growth models (SRGM) are applied to test failure data with the hope of obtaining a credible estimate of the software failure rate that will be observed in the field. Unfortunately, test environments are often different from field environments.

Within the context of a widely used SRGM, we develop a likelihood ratio test to formally determine if the test environment and the field environment are similar. When the environments differ, we propose the use of calibration factors to compensate for the mismatch. If projects have both test and field failure data for a previous release, the calibration factors can be empirically evaluated and used to estimate the field failure rate of a subsequent release. For a new project, matching the software's function and architecture to projects with both test and field failure data can provide semi-empirical calibration factors. We illustrate the methodology by applying it to a telecommunications software system.