Abstract

Transformer-based encoder-decoder models produce a fused token-wiserepresentation after every encoder layer. We investigate the effects ofallowing the encoder to preserve and explore alternative hypotheses, combinedat the end of the encoding process. To that end, we design and examine a$\textit{Multi-stream Transformer}$ architecture and find that splitting theTransformer encoder into multiple encoder streams and allowing the model tomerge multiple representational hypotheses improves performance, with furtherimprovement obtained by adding a skip connection between the first and thefinal encoder layer.

 

Quick Read (beta)