Abstract:

Classifiers are often expected for estimating the classification probabilities (CP) besides predicting classification labels. However, many classifiers output biased CP estimates due to misspecified models, problematic modelfittings, or improper CPestimating algorithms. Two popular postfitting methods for CP calibration are the Platt scaling and the isotonic regression. This work evaluated the CP performance for the two methods for the following four classifiers: Support Vector Machine (SVM) Boosted Decision Tree (GB) Random Forest (RF) Neural Network (NN) The overall CP performance are measured by using the logloss score, Brier's score, area under the ROC curve, classification accuracy, precision, recall, and Fmeasure. This study used SAS Software to fit all the four classifiers and measure their relevant CP performance. Our empirical results confirm that Platt scaling is more accurate than isotonic regression if the intensityCP distortion is a sigmoid function, otherwise isotonic regression is more flexible for correcting any monotonic distortion.
