Using Bootstraps to Evaluate the Effect of Sample Size Reduction on Scoring System Performance

• Activity Number: 107
• Type: Contributed
• Date/Time: Monday, August 12, 2002 : 10:30 AM to 12:20 PM
• Sponsor: Section on Survey Research Methods*
• Author(s): William Wong*+ and Chih-Chin Ho+
• Affiliation(s): Internal Revenue Service and Internal Revenue Service
• Keywords: Balanced Bootstrap ; Resampling ; Nonlinear Estimator ; Variance Estimation ; Classification ; Restratification

Abstract

Currently, the U. S. Internal Revenue Service calculates a scoring formula for each return and uses it as one criterion to determine which returns to audit. Periodically, IRS updates this formula from a stratified random audit sample. In 1988, such an audit sample was selected. The sample was used to derive a new scoring formula. Each sample return was then scored and sorted by descending scores. The top x%, say 5%, of the sample is then evaluated to determine the "hit rate," the percentage of that 5% of the returns that were "profitable to audit."

The issue was raised that if we reduced the sample by 20% or more, how would the "hit rate" be affected. Since the "hit rate" is a function of the "black box," it is a nonlinear estimator of unknown characteristics. To analyze the effect of sample reduction on such a scoring system, we calculate balanced bootstrap estimates, their differences, and their variances. This paper presents the design and the results of this analysis.