|
In classical hypothesis testing of a single hypothesis one looks for desirable properties based on size and power. For example, one might search for most powerful tests among unbiased or invariant test procedures. In one or two dimensions it is usually possible to make a sketch and get a good look at acceptance and rejection regions. As the number of hypotheses grows procedures become more complex and the optimality goals change. We tend to talk about things like the total number of mistakes, the Familywise Error Rate and the False Discovery Rate. These are still in the spirit of size and power. For very large numbers of hypotheses asymptotic considerations are also reasonable.
There is another type of issue that may be of concern. We will see that for many common procedures they have the potential to result in “politically incorrect” decisions if used more than once. That is, Outcome A is given higher significance than Outcome B when it was clearly the intent of the researcher to have the reverse. The question is what, if anything, can and should be done about it?
|
