SUPPORTIVE ATTENTION IN END-TO-END MEMORY NETWORKS

Abstract

Attention over natural language aims to spotlight on the meaningful region with representative keywords which can extract desirable meanings to accomplish a task of interest. The attention parameter is a latent variable which was indirectly estimated by minimizing the classification loss. For the task of question answering (QA), the classification loss may not sufficiently reflect the target answer. This paper proposes a direct solution which attends the meaningful region by minimizing the reconstruction loss due to auxiliary or supporting data which are available in different scenarios. In particular, minimizing the classification and reconstruction losses are carried out under the end-to-end memory network so that the memory-augmented question answering is realized. Such a supportive attention is implemented as a sequence-tosequence model which reconstructs the supporting sentence to assure the translation invariance. The merit of this method is sufficiently demonstrated for sequential learning by using the bAbI QA and dialog tasks.