Abstract
Deep learning achieves state-of-the-art results in many areas. However recentworks have shown that deep networks can be vulnerable to adversarialperturbations which slightly changes the input but leads to incorrectprediction. Adversarial training is an effective way of improving therobustness to the adversarial examples, typically formulated as a robustoptimization problem for network training. To solve it, previous works directlyrun gradient descent on the "adversarial loss", i.e. replacing the input datawith the corresponding adversaries. A major drawback of this approach is thecomputational overhead of adversary generation, which is much larger thannetwork updating and leads to inconvenience in adversarial defense. To address this issue, we fully exploit structure of deep neural networks andpropose a novel strategy to decouple the adversary update with the gradientback propagation. To achieve this goal, we follow the research line consideringtraining deep neural network as an optimal control problem. We formulate therobust optimization as a differential game. This allows us to figure out thenecessary conditions for optimality. In this way, we train the neural networkvia solving the Pontryagin's Maximum Principle (PMP). The adversary is onlycoupled with the first layer weight in PMP. It inspires us to split theadversary computation from the back propagation gradient computation. As aresult, our proposed YOPO (You Only Propagate Once) avoids forward and backwardthe data too many times in one iteration, and restricts core descent directionscomputation to the first layer of the network, thus speeding up every iterationsignificantly. For adversarial example defense, our experiment shows that YOPOcan achieve comparable defense accuracy using around 1/5 GPU time of theoriginal projected gradient descent training.