Improving Likelihood-based Data-squashing

• Activity Number: 425
• Type: Contributed
• Date/Time: Thursday, August 12, 2004 : 10:30 AM to 12:20 PM
• Sponsor: Section on Statistical Computing
• Author(s): Yuchung J. Wang*+ and Xiaodong Sun
• Companies: Rutgers University-Camden and Ask Jeeves, Inc.
• Keywords: data mining ; imputation ; uniform design ; clustering ; representative observation

Abstract

Data-squashing is reducing a very large dataset, called the mother data, into a much smaller dataset, called a squashed data, with the requirement that statistical models/decisions derived from a squashed data are identical/similar to those derived from the mother data. A squashed data may be viewed as a collection of imputations. Madigan et al. (2003) proposed a likelihood-based squashing, which assumed a certain likelihood is the data-generating mechanism of the mother data. They artificially crate a likelihood profile for every observation in the mother data and subsequently cluster the mother data according to their profiles. Out of each cluster, an imputed observation will be constructed along with a weight. We address the issue: how to construct better profile. We shall illustrate that (a) the more the likelihood computed, the better the imputation, and (b) uniform design can improve upon factorial design in sampling the likelihood. We also propose an accelerated algorithm to perform clustering, because the k-mean clustering converges too slowly to be computationally feasible when the number of profiles is large.