Abstract
We present a unified representation of the most popular neural networkactivation functions. Adopting Mittag-Leffler functions of fractional calculus,we propose a flexible and compact functional form that is able to interpolatebetween various activation functions and mitigate common problems in trainingneural networks such as vanishing and exploding gradients. The presented gatedrepresentation extends the scope of fixed-shape activation functions to theiradaptive counterparts whose shape can be learnt from the training data. Thederivatives of the proposed functional form can also be expressed in terms ofMittag-Leffler functions making it a suitable candidate for gradient-basedbackpropagation algorithms. By training multiple neural networks of differentcomplexities on various datasets with different sizes, we demonstrate thatadopting a unified gated representation of activation functions offers apromising and affordable alternative to individual built-in implementations ofactivation functions in conventional machine learning frameworks.