The back-propagation algorithm is the cornerstone of deep learning. Despiteits importance, few variations of the algorithm have been attempted. This workpresents an approach to discover new variations of the back-propagationequation. We use a domain specific lan- guage to describe update equations as alist of primitive functions. An evolution-based method is used to discover newpropagation rules that maximize the generalization per- formance after a fewepochs of training. We find several update equations that can train faster withshort training times than standard back-propagation, and perform similar asstandard back-propagation at convergence.