Abstract
Basic binary relations such as equality and inequality are fundamental torelational data structures. Neural networks should learn such relations andgeneralise to new unseen data. We show in this study, however, that thisgeneralisation fails with standard feed-forward networks on binary vectors.Even when trained with maximal training data, standard networks do not reliablydetect equality.We introduce differential rectifier (DR) units that we add tothe network in different configurations. The DR units create an inductive biasin the networks, so that they do learn to generalise, even from small numbersof examples and we have not found any negative effect of their inclusion in thenetwork. Given the fundamental nature of these relations, we hypothesize thatfeed-forward neural network learning benefits from inductive bias in otherrelations as well. Consequently, the further development of suitable inductivebiases will be beneficial to many tasks in relational learning with neuralnetworks.