RECALL NEURAL NETWORK FOR SOURCE SEPARATION

Abstract

This paper presents a novel memory-augmented neural network for single-channel source separation. We propose a recall neural network (RCNN) where a couple of external memories are realized for sequence-to-sequence learning based on an encoder and a decoder. These memories are learned in a two-pass sensing procedure where the mixed signal is encoded and then decoded (or recalled) as context vectors by using a bidirectional long short-term memory (LSTM) and a LSTM, respectively. These context vectors are integrated in a gating layer. A set of attention weights are calculated to attend the hidden state of decoder to implement a recurrent neural network for source separation. A gated attention mechanism is carried out to fulfill a specialized memory network. The regression errors due to two passes of sensing procedure and one pass of gated attention are jointly minimized to estimate the weight parameters of different components in different layers. The experiments on multi-speaker speech enhancement show that the proposed RCNN consistently outperforms LSTM and neural Turing machine in different settings.