Prequential Tests in Classification and Regression Trees

• Activity Number: 383
• Type: Contributed
• Date/Time: Wednesday, August 11, 2004 : 2:00 PM to 3:50 PM
• Sponsor: Biometrics Section
• Author(s): Jialu Zhang*+ and Francoise Seillier-Moiseiwitsch
• Companies: University of Maryland, Baltimore County and University of Maryland, Baltimore County
• Keywords: prequential test ; classification ; CART ; cross-validation

Abstract

The data-mining procedure implemented in CART (Classification and Regression Trees) extracts information from the data by constructing binary decision trees. The current CART algorithm exhaustively searches all levels of all covariates and decides on the node-splitting criterion by minimizing node impurity. The node-splitting process goes on until there are too few observations at a node or until the node consists of observations of a single type. Then the large binary tree is pruned back using cross-validation or some other pruning approach. We implement into CART a new model evaluation method, based on prequential testing, to replace the splitting and pruning steps. As a direct consequence, these steps now involve formal statistical tests to validate the decisions. This approach is suited to both parametric and nonparametric decision rules, and assesses models on the basis of the accuracy of their probabilistic predictions for future events. More specifically, data are first divided into a training sample and an evaluation sample. The training sample is extended by one observation at a time to update estimates and make a prediction for the next outcome.