Abstract
Transformers achieve state-of-the-art accuracy and robustness across manytasks, but an understanding of their inductive biases and how those biasesdiffer from other neural network architectures remains elusive. In this work,we identify the sensitivity of the model to token-wise random perturbations inthe input as a unified metric which explains the inductive bias of transformersacross different data modalities and distinguishes them from otherarchitectures. We show that transformers have lower sensitivity than MLPs,CNNs, ConvMixers and LSTMs, across both vision and language tasks. We also showthat this low-sensitivity bias has important implications: i) lower sensitivitycorrelates with improved robustness; it can also be used as an efficientintervention to further improve the robustness of transformers; ii) itcorresponds to flatter minima in the loss landscape; and iii) it can serve as aprogress measure for grokking. We support these findings with theoreticalresults showing (weak) spectral bias of transformers in the NTK regime, andimproved robustness due to the lower sensitivity. The code is available athttps://github.com/estija/sensitivity.