Consistency and Generalization Error Bound of Neural Network Models

• Activity Number: 378
• Type: Contributed
• Date/Time: Wednesday, August 11, 2004 : 2:00 PM to 3:50 PM
• Sponsor: General Methodology
• Author(s): Indrias G. Berhane*+
• Companies: University of Kentucky
• Keywords: generalization error ; empirical mean square error ; uniform stability ; smoothing regularization

Abstract

Estimation using flexible models based on finite and noisy datasets is a challenging task. The main concern in using flexible models is the possibility of overfitting the training data. Feed-forward neural networks are class of approximating functions with a universal approximation capabilities. For a class of functions with known approximation capabilities the main concern is performance on future datasets. In real-world applications the availability of training samples are often limited and noisy. Since it is possible to find a neural network model which minimizes the empirical error but may not minimize the generalization error we restrict our focus to stable models only. We use a smoothing regularization as a method of stabilizer. We provide an upper bound of the generalization error of a neural network, estimated with a smoothing regularization. We also prove possible generalization error of a neural network estimate converges to the minimum achievable by the class as the training sample increases, with high probability. Our numerical experiments show neural network models estimated with smoothing regularization give similar or less prediction errors.