|
Abstract:
|
Many statistical models rely on assumptions that, when violated, threaten their substantive conclusions. Motivated by a desire to demonstrate robustness, researchers may test the null hypothesis of "no violation" of these assumptions. Failing to find evidence to reject this null, they may conclude that the assumption holds, especially if the point estimate of the violation is small. However, this approach inappropriately reverses the importance of Type I and Type II errors. In many cases, it may miss important violations due to lack of sufficient statistical power or, with adequate statistical power, detect violations that are "statistically significant" but practically trivial. Using the popular difference-in-differences setting, we reformulate model assumption tests in a non-inferiority framework and focus on ruling out violations that would meaningfully change main effect estimates. We conclude that ruling out meaningful violations of modeling assumptions requires additional statistical power, often similar in magnitude to adoption of more robust models.
|
