Abstract
Successful deep learning models often involve training neural networkarchitectures that contain more parameters than the number of training samples.Such overparametrized models have been extensively studied in recent years, andthe virtues of overparametrization have been established from both thestatistical perspective, via the double-descent phenomenon, and thecomputational perspective via the structural properties of the optimizationlandscape. Despite the remarkable success of deep learning architectures in theoverparametrized regime, it is also well known that these models are highlyvulnerable to small adversarial perturbations in their inputs. Even whenadversarially trained, their performance on perturbed inputs (robustgeneralization) is considerably worse than their best attainable performance onbenign inputs (standard generalization). It is thus imperative to understandhow overparametrization fundamentally affects robustness. In this paper, we will provide a precise characterization of the role ofoverparametrization on robustness by focusing on random features regressionmodels (two-layer neural networks with random first layer weights). We considera regime where the sample size, the input dimension and the number ofparameters grow in proportion to each other, and derive an asymptotically exactformula for the robust generalization error when the model is adversariallytrained. Our developed theory reveals the nontrivial effect ofoverparametrization on robustness and indicates that for adversarially trainedrandom features models, high overparametrization can hurt robustgeneralization.