Abstract:
|
We propose a new framework for conducting Monte Carlo simulations to test statistical estimation methods using the Python library Signac. This approach to simulation allows statisticians to conduct large-scale exercises to test their method by drawing repeated samples across a large number of parameter sweeps. We demonstrate how the multidimensional indexable storage layout offered by Signac is particularly suited to conduct diagnostics on statistical methods and identify issues ranging from identifiability to detection of issues in in-built subroutines. We use this framework to test Bayesian and frequentist estimation algorithms for a latent class model. In a Bayesian setting, this framework allows for the storage of large posterior samples for each sample draw and for each combination of parameters across the hierarchical structure, and can be extended to conduct prior sensitivity. We consider 8,000 parameter combinations and design a series of aggregation methods to utilize the rich array of simulation output and evaluate the performance of the estimation algorithms.
|