Abstract
Continuous sentence embeddings using recurrent neural networks (RNNs), wherevariable-length sentences are encoded into fixed-dimensional vectors, are oftenthe main building blocks of architectures applied to language tasks such asdialogue generation. While it is known that those embeddings are able to learnsome structures of language (e.g. grammar) in a purely data-driven manner,there is very little work on the objective evaluation of their ability to coverthe whole language space and to generalize to sentences outside the languagebias of the training data. Using a manually designed context-free grammar (CFG)to generate a large-scale dataset of sentences related to the content ofrealistic 3D indoor scenes, we evaluate the language coverage andgeneralization abilities of the most common continuous sentence embeddingsbased on RNNs. We also propose a new embedding method based on arithmeticcoding, AriEL, that is not data-driven and that efficiently encodes incontinuous space any sentence from the CFG. We find that RNN-based embeddingsunderfit the training data and cover only a small subset of the languagedefined by the CFG. They also fail to learn the underlying CFG and generalizeto unbiased sentences from that same CFG. We found that AriEL provides aninsightful baseline.