Abstract:
|
Noise injection (NI) is a method to mitigate overfitting in neural networks (NNs). The recent developments in Bernoulli NI as implemented in dropout and shakeout demonstrate the efficiency and feasibility of NI in regularizing NNs. We propose whiteout, a new regularization technique via adaptive Gaussian NI in deep NNs. Whiteout is associated with a closed-form penalized objective function in GLMs that includes bridge, (adaptive) lasso, ridge regression, and elastic net as special cases. Whiteout can also be viewed as robust learning of NN model in the presence of small perturbations in input and hidden nodes. The noise-perturbed empirical loss function with whiteout converges almost surely to the ideal loss function, and the estimates of NN parameters obtained from minimizing the former are consistent with those obtained from minimizing the idea loss function. Whiteout performs better in small training data compared to dropout, and its objective function is more stable compared to shakeout.
|