Abstract

Softmax is the de facto standard in modern neural networks for languageprocessing when it comes to normalizing logits. However, by producing a denseprobability distribution each token in the vocabulary has a nonzero chance ofbeing selected at each generation step, leading to a variety of reportedproblems in text generation. $\alpha$-entmax of arXiv:1905.05702 solves thisproblem, but is considerably slower than softmax. In this paper, we propose an alternative to $\alpha$-entmax, which keeps itsvirtuous characteristics, but is as fast as optimized softmax and achieves onpar or better performance in machine translation task.

 

Quick Read (beta)