Abstract
This paper proposes a new architecture - Attentive Tensor Product Learning(ATPL) - to represent grammatical structures in deep learning models. ATPL is anew architecture to bridge this gap by exploiting Tensor ProductRepresentations (TPR), a structured neural-symbolic model developed incognitive science, aiming to integrate deep learning with explicit languagestructures and rules. The key ideas of ATPL are: 1) unsupervised learning ofrole-unbinding vectors of words via TPR-based deep neural network; 2) employingattention modules to compute TPR; and 3) integration of TPR with typical deeplearning architectures including Long Short-Term Memory (LSTM) and FeedforwardNeural Network (FFNN). The novelty of our approach lies in its ability toextract the grammatical structure of a sentence by using role-unbindingvectors, which are obtained in an unsupervised manner. This ATPL approach isapplied to 1) image captioning, 2) part of speech (POS) tagging, and 3)constituency parsing of a sentence. Experimental results demonstrate theeffectiveness of the proposed approach.