414 – Model Building and Selection
Number of Samples Needed for Model Selection with Confidence
Sean G. Carver
American University
A common measure used to quantify the similarity of two models is the Kullback-Leibler divergence, computed from a true model to an alternative model. We propose a different measure: the number of samples needed to correctly reject the alternative model with a given confidence level (e.g. 95%). Our method works as follows: (1) we simulate samples from the true model, (2) for each sample, we compute a log-likelihood ratio (3), we bootstrap and sum the log-likelihood ratios---when this sum is positive, we select the true model, (4) using simple linear regression, we determine the number of terms (i.e. number of samples) needed to make the desired quantile (e.g. 5%) fall at zero. We have tested this method on t-distributions of different degrees of freedom and have confirmed that it gives reasonably consistent results. However, we plan to apply this method to Markov chains, e.g. used for sports statistics like tennis, volleyball, and baseball. For these applications, it may be desirable to have a measure that is easier to interpret than the Kullback-Leibler divergence. How many innings are needed to falsify the model of the Yankees when simulating a model of the Orioles?