|
Abstract:
|
This paper discusses a comparison among Bayesian statistical inference, maximum likelihood estimation, and maximum entropy methods. Using a toy example we compare estimation results using maximum entropy as defined by Jaynes, Bayesian inference, and the maximum likelihood method. This example discusses the relationship among those methods and highlights the differences and similarities among them. The Bayesian inference example is developed using some form of prior knowledge, the maximum entropy solution is derived from suitable constraints, while maximum likelihood maximizes sensitivity to the observed data. The results that correspond to such methods are discussed from the point of view of information theory and a statistical approach by comparing related quantities. For instance, a maximum likelihood solution can be made equivalent to a maximum entropy estimation and Bayesian inference under certain conditions. Finally, some conclusions are drawn regarding the information gain and Kullback-Leibler divergence when updating knowledge from the prior to the posterior in Bayesian inference, its relationship with maximum entropy estimation and maximum likelihood method.
|
