Abstract
Recent works have highlighted the strengths of the Transformer architecturefor dealing with sequence tasks. At the same time, neural architecture searchhas advanced to the point where it can outperform human-designed models. Thegoal of this work is to use architecture search to find a better Transformerarchitecture. We first construct a large search space inspired by the recentadvances in feed-forward sequential models and then run evolutionaryarchitecture search, seeding our initial population with the Transformer. Toeffectively run this search on the computationally expensive WMT 2014English-German translation task, we develop the progressive dynamic hurdlesmethod, which allows us to dynamically allocate more resources to morepromising candidate models. The architecture found in our experiments - theEvolved Transformer - demonstrates consistent improvement over the Transformeron four well-established language tasks: WMT 2014 English-German, WMT 2014English-French, WMT 2014 English-Czech and LM1B. At big model size, the EvolvedTransformer is twice as efficient as the Transformer in FLOPS without loss inquality. At a much smaller - mobile-friendly - model size of ~7M parameters,the Evolved Transformer outperforms the Transformer by 0.7 BLEU on WMT'14English-German.