Abstract
In this paper, we propose a novel implicit semantic data augmentation (ISDA)approach to complement traditional augmentation techniques like flipping,translation or rotation. Our work is motivated by the intriguing property thatdeep networks are surprisingly good at linearizing features, such that certaindirections in the deep feature space correspond to meaningful semantictransformations, e.g., adding sunglasses or changing backgrounds. As aconsequence, translating training samples along many semantic directions in thefeature space can effectively augment the dataset to improve generalization. Toimplement this idea effectively and efficiently, we first perform an onlineestimate of the covariance matrix of deep features for each class, whichcaptures the intra-class semantic variations. Then random vectors are drawnfrom a zero-mean normal distribution with the estimated covariance to augmentthe training data in that class. Importantly, instead of augmenting the samplesexplicitly, we can directly minimize an upper bound of the expectedcross-entropy (CE) loss on the augmented training set, leading to a highlyefficient algorithm. In fact, we show that the proposed ISDA amounts tominimizing a novel robust CE loss, which adds negligible extra computationalcost to a normal training procedure. Although being simple, ISDA consistentlyimproves the generalization performance of popular deep models (ResNets andDenseNets) on a variety of datasets, e.g., CIFAR-10, CIFAR-100 and ImageNet.Code for reproducing our results are available athttps://github.com/blackfeather-wang/ISDA-for-Deep-Networks.