Abstract
Sequence-to-sequence models have been widely used in end-to-end speechprocessing, for example, automatic speech recognition (ASR), speech translation(ST), and text-to-speech (TTS). This paper focuses on an emergentsequence-to-sequence model called Transformer, which achieves state-of-the-artperformance in neural machine translation and other natural language processingapplications. We undertook intensive studies in which we experimentallycompared and analyzed Transformer and conventional recurrent neural networks(RNN) in a total of 15 ASR, one multilingual ASR, one ST, and two TTSbenchmarks. Our experiments revealed various training tips and significantperformance benefits obtained with Transformer for each task including thesurprising superiority of Transformer in 13/15 ASR benchmarks in comparisonwith RNN. We are preparing to release Kaldi-style reproducible recipes usingopen source and publicly available datasets for all the ASR, ST, and TTS tasksfor the community to succeed our exciting outcomes.