Abstract
Activation functions play a pivotal role in the function learning usingneural networks. The non-linearity in the learned function is achieved byrepeated use of the activation function. Over the years, numerous activationfunctions have been proposed to improve accuracy in several tasks. Basicfunctions like ReLU, Exponential, Tanh, or Softplus have been favorite amongthe deep learning community because of their simplicity. In recent years,several novel activation functions arising from these basic functions have beenproposed, which have improved accuracy in some challenging datasets withcomplicated models. We propose a five hyper-parameters family of activationfunctions, namely EIS, defined as, \[\frac{x(\ln(1+e^x))^\alpha}{\sqrt{\beta+\gamma x^2}+\delta e^{-\theta x}}. \]We show examples of activation functions from the EIS family which outperformwidely used activation functions on some well known datasets and models. Forexample, $\frac{x\ln(1+e^x)}{x+1.16e^{-x}}$ beats ReLU by 0.89\% inDenseNet-169, 0.24\% in Inception V3 in CIFAR100 dataset while 1.13\% inInception V3, 0.13\% in DenseNet-169, 0.94\% in SimpleNet model in CIFAR10dataset. Also, $\frac{x\ln(1+e^x)}{\sqrt{1+x^2}}$ beats ReLU by 1.68\% inDenseNet-169, 0.30\% in Inception V3 in CIFAR100 dataset while 1.0\% inInception V3, 0.15\% in DenseNet-169, 1.13\% in SimpleNet model in CIFAR10dataset