Improving Distribution Estimators Under Kullback-Leibler Loss

• Activity Number: 576
• Type: Contributed
• Date/Time: Wednesday, August 1, 2012 : 2:00 PM to 3:50 PM
• Sponsor: IMS
• Author(s): Qiang Wu*+ and Paul Vos
• Companies: East Carolina University and East Carolina University
• Keywords: KL loss ; Distribution unbiasedness ; Rao-Blackwell ; MVUE

Abstract

Estimators can often be modified to reduce their bias and/or variance, or to provide some other improvement in terms of the parameter being estimated. The Rao-Blackwell theorem can be used in this regard and minimum variance unbiased estimators are a well-known result of the Lehmann-Scheffe theorem. However, when the entire distribution is of interest, rather than just a particular feature associated with the parameter, parameter-invariance is a desirable property. In this paper, we employ a parameter-free distribution estimation framework and utilize the Kullback-Leibler (KL) divergence as a loss function. A distributional version of the Rao-Blackwell theorem is given under the KL loss. Wu and Vos (2011) show that the KL risk of a distribution estimator decomposes in a parallel fashion as the mean squared error decomposition for a parameter estimator, and that a distribution estimator is unbiased iff its distribution mean is equal to the true distribution. We show that for exponential families the maximum likelihood estimator is the unique minimum KL risk unbiased estimator. Illustrative examples are provided.