Abstract
Neural networks (NNs) can learn to rely on spurious signals in the trainingdata, leading to poor generalisation. Recent methods tackle this problem bytraining NNs with additional ground-truth annotations of such signals. Thesemethods may, however, let spurious signals re-emerge in deep convolutional NNs(CNNs). We propose Targeted Activation Penalty (TAP), a new method tackling thesame problem by penalising activations to control the re-emergence of spurioussignals in deep CNNs, while also lowering training times and memory usage. Inaddition, ground-truth annotations can be expensive to obtain. We show that TAPstill works well with annotations generated by pre-trained models as effectivesubstitutes of ground-truth annotations. We demonstrate the power of TAPagainst two state-of-the-art baselines on the MNIST benchmark and on twoclinical image datasets, using four different CNN architectures.