Keywords: Model evaluation, statistical uncertainty, parameter interpretation, fungible estimates
When evaluating statistical models, data scientists must make decisions regarding the credibility of results. This task is often made more difficult when a goal of the analysis is explanatory modeling (i.e., not solely predictive) and experimental data is not available to justify causality. In this case, observational data along with theory, model-fit (e.g., RMSE), and other statistical techniques (e.g., sensitivity analysis, causal search) must be used to build a case for the plausibility of a given models results. Previous research has indicated that marginal decreases in data-model fit can produce a range of equally fitting (i.e., fungible) parameter estimates that, at times, may lead to very different interpretations. Whereas standard errors represent a measure of uncertainty tied to sampling variability, fungible parameter estimates describe a distinct type of uncertainty related to model fit. An examination of fungible parameter estimates, presents information that can be used to either strengthen or weaken subsequent interpretation of parameter estimates. Methods for exploring parameter stability relative to small decrements in model fit have been recommended for several models (e.g., multiple regression, logistic regression, path analysis). However, these methods are limited and are not implemented in software. The current project utilizes a simulated annealing algorithm as a novel method for exploring the fungible parameter space. Finally, the proposed method is implemented in an R package that allows researchers to visually examine the stability of parameter estimates in their own results. Practical examples, along with guidelines for interpretation will also be included.