Abstract
Neural attention has become central to many state-of-the-art models innatural language processing and related domains. Attention networks are aneasy-to-train and effective method for softly simulating alignment; however,the approach does not marginalize over latent alignments in a probabilisticsense. This property makes it difficult to compare attention to other alignmentapproaches, to compose it with probabilistic models, and to perform posteriorinference conditioned on observed data. A related latent approach, hardattention, fixes these issues, but is generally harder to train and lessaccurate. This work considers variational attention networks, alternatives tosoft and hard attention for learning latent variable alignment models, withtighter approximation bounds based on amortized variational inference. Wefurther propose methods for reducing the variance of gradients to make theseapproaches computationally feasible. Experiments show that for machinetranslation and visual question answering, inefficient exact latent variablemodels outperform standard neural attention, but these gains go away when usinghard attention based training. On the other hand, variational attention retainsmost of the performance gain but with training speed comparable to neuralattention.