Abstract
We propose a neural language model capable of unsupervised syntacticstructure induction. The model leverages the structure information to formbetter semantic representations and better language modeling. Standardrecurrent neural networks are limited by their structure and fail toefficiently use syntactic information. On the other hand, tree-structuredrecursive networks usually require additional structural supervision at thecost of human expert annotation. In this paper, We propose a novel neurallanguage model, called the Parsing-Reading-Predict Networks (PRPN), that cansimultaneously induce the syntactic structure from unannotated sentences andleverage the inferred structure to learn a better language model. In our model,the gradient can be directly back-propagated from the language model loss intothe neural parsing network. Experiments show that the proposed model candiscover the underlying syntactic structure and achieve state-of-the-artperformance on word/character-level language model tasks.