|
Abstract:
|
Statistical tests--comparing a p-value to a prespecified threshold--are ubiquitous. While their use is controversial, in some areas the binary decisions they produce are appealingly simple and reflect how knowledge will be acquired (e.g. economic policy changes, drug regulation). However, assessing a test's reliability post hoc is notoriously difficult; existing methods are prone to paradoxical results, misinterpretation, dependence on further arbitrary thresholds, and/or have failed to gain popularity. As an alternative we consider a decision-theoretic approach to testing, that produces natural analogs of one- and two-sided significance tests. Applying ``loss estimation'' (as well as risk estimation) to these decisions, we derive novel measures of test reliability. The feasibility, accuracy, precision, and usefulness of these methods will be presented, and they will be compared to existing methods.
|
