On the Suboptimality of the MLE and All Other Point Estimators When the Model Is Wrong and Their Improvement via Flattening

• Activity Number: 223
• Type: Topic Contributed
• Date/Time: Monday, August 1, 2011 : 2:00 PM to 3:50 PM
• Sponsor: Section on Statistical Learning and Data Mining
• Author(s): Wojciech Kotlowski*+ and Peter Grünwald and Steven de Rooij
• Companies: Centrum Wiskunde & Informatica (CWI) and Centrum Wiskunde & Informatica (CWI) and Centrum Wiskunde & Informatica (CWI)
• Keywords: exponential family ; KL-risk ; prequential analysis ; model misspecification ; flattened maximum likelihood

Abstract

We analyze the KL-risk of point estimators for density estimation with exponential family models. It is known that when the data are generated by one of the distributions in the model, the maximum likelihood estimator (MLE) admits the optimal rate of convergence with the optimal constant factor in front, but when the model is misspecified (data comes from a distribution outside the model), MLE behaves suboptimally. We show that in the misspecified case, not only MLE, but every point estimator is unable to achieve the optimal constant in front of the convergence rate; the additional factor can become arbitrarily large. We then provide a solution to this problem by introducing a simple ``flattening' of the ML distribution, that does achieve the optimal performance even when the model is wrong. The flattened ML distribution lies slightly outside the model under consideration, and is thus not a point estimator. We can apply the flattened ML estimator in Dawid's prequential model selection criterion. Simulations show a major improvement of the resulting model selection criterion over standard prequential model selection based on MLE.