Abstract
In this paper, we show that standard feed-forward and recurrent neuralnetworks fail to learn abstract patterns based on identity rules. We proposeRepetition Based Pattern (RBP) extensions to neural network structures thatsolve this problem and answer, as well as raise, questions about integratingstructures for inductive bias into neural networks. Examples of abstractpatterns are the sequence patterns ABA and ABB where A or B can be any object.These were introduced by Marcus et al (1999) who also found that 7 month oldinfants recognise these patterns in sequences that use an unfamiliar vocabularywhile simple recurrent neural networks do not.This result has been contested inthe literature but it is confirmed by our experiments. We also show that theinability to generalise extends to different, previously untested, settings. We propose a new approach to modify standard neural network architectures,called Repetition Based Patterns (RBP) with different variants forclassification and prediction. Our experiments show that neural networks withthe appropriate RBP structure achieve perfect classification and predictionperformance on synthetic data, including mixed concrete and abstract patterns.RBP also improves neural network performance in experiments with real-worldsequence prediction tasks. We discuss these finding in terms of challenges forneural network models and identify consequences from this result in terms ofdeveloping inductive biases for neural network learning.