Abstract
Recurrent Neural Networks (RNNs) have been widely used in processing naturallanguage tasks and achieve huge success. Traditional RNNs usually treat eachtoken in a sentence uniformly and equally. However, this may miss the richsemantic structure information of a sentence, which is useful for understandingnatural languages. Since semantic structures such as word dependence patternsare not parameterized, it is a challenge to capture and leverage structureinformation. In this paper, we propose an improved variant of RNN,Multi-Channel RNN (MC-RNN), to dynamically capture and leverage local semanticstructure information. Concretely, MC-RNN contains multiple channels, each ofwhich represents a local dependence pattern at a time. An attention mechanismis introduced to combine these patterns at each step, according to the semanticinformation. Then we parameterize structure information by adaptively selectingthe most appropriate connection structures among channels. In this way, diverselocal structures and dependence patterns in sentences can be well captured byMC-RNN. To verify the effectiveness of MC-RNN, we conduct extensive experimentson typical natural language processing tasks, including neural machinetranslation, abstractive summarization, and language modeling. Experimentalresults on these tasks all show significant improvements of MC-RNN over currenttop systems.