Abstract
Sequence-to-sequence (Seq2seq) models have played an import role in therecent success of various natural language processing methods, such as machinetranslation, text summarization, and speech recognition. However, currentSeq2seq models have trouble preserving global latent information from a longsequence of words. Variational autoencoder (VAE) alleviates this problem bylearning a continuous semantic space of the input sentence. However, it doesnot solve the problem completely. In this paper, we propose a new recurrentneural network (RNN)-based Seq2seq model, RNN semantic variational autoencoder(RNN--SVAE), to better capture the global latent information of a sequence ofwords. To consider the words in a sentence equally, without regard to itsposition within the sentence, we construct a document information vector usingthe attention information between the final state of the encoder and everyprior hidden state. Then, we combine this document information vector with thefinal hidden state of the bi-directional RNN encoder to construct the globallatent vector, which becomes the output of the encoder part. Then, the mean andstandard deviation of the continuous semantic space are learned to takeadvantage of the variational method. Experimental results of three naturallanguage tasks (i.e., language modeling, missing word imputation, paraphraseidentification) confirm that the proposed RNN--SVAE yields higher performancethan two benchmark models.