Speeding Up Entmax

  • 2021-11-12 17:34:11
  • Maxat Tezekbayev, Vassilina Nikoulina, Matthias GallĂ©, Zhenisbek Assylbekov
  • 21

Abstract

Softmax is the de facto standard in modern neural networks for languageprocessing when it comes to normalizing logits. However, by producing a denseprobability distribution each token in the vocabulary has a nonzero chance ofbeing selected at each generation step, leading to a variety of reportedproblems in text generation. $\alpha$-entmax of arXiv:1905.05702 solves thisproblem, but is considerably slower than softmax. In this paper, we propose an alternative to $\alpha$-entmax, which keeps itsvirtuous characteristics, but is as fast as optimized softmax and achieves onpar or better performance in machine translation task.

 

Quick Read (beta)

loading the full paper ...