|
Activity Number:
|
520
|
|
Type:
|
Invited
|
|
Date/Time:
|
Thursday, August 10, 2006 : 10:30 AM to 12:20 PM
|
|
Sponsor:
|
Classification Society of North America
|
| Abstract - #305232 |
|
Title:
|
Assessing the Risk of Classification Decisions
|
|
Author(s):
|
Roman D. Fresnedo*+ and Dragos D. Margineantu
|
|
Companies:
|
The Boeing Company and The Boeing Company
|
|
Address:
|
M&CT, Applied Statistics, Seattle, WA, 98124-2207,
|
|
Keywords:
|
classification ; risk ; non-uniform cost ; hypothesis testing ; bootstrap ; Dirichlet priors
|
|
Abstract:
|
Most learning algorithms are not amenable to complete mathematical analysis. In spite of proven performance, proofs and constructive theories on how to choose classifier parameters or assess performance are rare. We present and compare bootstrap and Bayesian methods for assessing the expected risk of single classifiers' decisions, and analyze the estimation and correction of small values in the confusion matrix. We extend our techniques to the paired comparison of classifiers, of importance in industrial applications. How much evidence is there in favor of one classifier over another? What are the right criteria and metrics to compare classification decisions? Preliminary experiments show that the correction of small entries with uniform priors can lead to large errors in the estimated risk. We propose and study alternatives.
|
