|
Abstract:
|
Summary measures of classification performance are used widely in the tasks related to classification, prediction, decision making, and classifier optimization. They include such generally used summary statistics as odds ratio and kappa coefficient, summary indices well-known in the decision making field such as Youden’s index, as well as more technical indices such as proximity to the perfect classifier, misclassification rate, etc. In addition to the classic summary measures, new performance indices are continued to be regularly proposed to emphasize particular characteristics of a targeted classification task. We demonstrate that some of the popular summary indices have intrinsic ability to lead to objective errors in ranking classifiers. We develop a formal concept of “improper” functional summary of classification performance, and provide a practical criterion for evaluating summary indices. The concept and tools are then used to determine the proper/improper nature common summary indices. The developed framework enables identifying summaries indices that should be used with caution, and provides guidelines for development of robust approaches for ranking classifiers.
|
