TIKHONOV REGULARIZATION FOR DEEP NEURAL NETWORK ACOUSTIC MODELING

Abstract

Deep neural network (DNN) has been widely demonstrated to achieve high performance in different speech recognition tasks. This paper focuses on the issue of model regularization in DNN acoustic model. Our idea is to compensate for the perturbations over training samples in the restricted Boltzmann machine (RBM) which is applied as a pre-training stage for unsupervised feature learning and structural modeling. We introduce the Tikhonov regularization in pre-training procedure and pursue the invariance property of objective function over the variations in input samples. This Tikhonov regularization is further combined with the regularization based on weight decay. The error function in supervised cross-entropy training is accordingly reduced. Experimental results on using RM and Aurora4 tasks show that hybrid regularization in RBM pre-training improves the training condition in DNN acoustic model and the robustness in speech recognition performance.