Online Program

Model Evaluation Metrics (304157)

Once a model is employed it is useful to know how well it performs. There are several model performance metrics available. Conclusions made about a model’s performance using a variety of evaluation metrics demonstrated consistent conclusions when applied to inventory models. Described herein are the Receiver Operator Characteristics (ROC), Area Under the Curve (AUC), model score distribution, predicted vs. actual results including the absolute error rate, Kolmogorov-Smirnov (KS) statistic, and the Lorenz curve and Gini coefficient. Applied to inventory these metrics identify which inventory models are performing well and which should be reexamined. Applicability to Excise tax inventory is discussed. Models evaluated consisted of 46 binomial logit models to score potential inventory on a weekly basis. The dependent variables involved whether or not a case achieved a certain outcome based on a treatment and within a defined timeframe. For example, a full pay outcome in the call center treatment stream within 30 weeks. The model evaluation metrics applied to inventory demonstrated consistent conclusions. A similar approach is recommended for the Excise tax inventory models where the dependent variable is how likely an Excise tax case is of having an adjustment.