Abstract
Attention mechanism has been used as an ancillary means to help RNN or CNN.However, the Transformer (Vaswani et al., 2017) recently recorded thestate-of-the-art performance in machine translation with a dramatic reductionin training time by solely using attention. Motivated by the Transformer,Directional Self Attention Network (Shen et al., 2017), a fully attention-basedsentence encoder, was proposed. It showed good performance with various data byusing forward and backward directional information in a sentence. But in theirstudy, not considered at all was the distance between words, an importantfeature when learning the local dependency to help understand the context ofinput text. We propose Distance-based Self-Attention Network, which considersthe word distance by using a simple distance mask in order to model the localdependency without losing the ability of modeling global dependency whichattention has inherent. Our model shows good performance with NLI data, and itrecords the new state-of-the-art result with SNLI data. Additionally, we showthat our model has a strength in long sentences or documents.