Abstract
Black-box adversarial attacks present a realistic threat to actionrecognition systems. Existing black-box attacks follow either a query-basedapproach where an attack is optimized by querying the target model, or atransfer-based approach where attacks are generated using a substitute model.While these methods can achieve decent fooling rates, the former tends to behighly query-inefficient while the latter assumes extensive knowledge of theblack-box model's training data. In this paper, we propose a new attack onaction recognition that addresses these shortcomings by generatingperturbations to disrupt the features learned by a pre-trained substitute modelto reduce the number of queries. By using a nearly disjoint dataset to trainthe substitute model, our method removes the requirement that the substitutemodel be trained using the same dataset as the target model, and leveragesqueries to the target model to retain the fooling rate benefits provided byquery-based methods. This ultimately results in attacks which are moretransferable than conventional black-box attacks. Through extensiveexperiments, we demonstrate highly query-efficient black-box attacks with theproposed framework. Our method achieves 8% and 12% higher deception ratescompared to state-of-the-art query-based and transfer-based attacks,respectively.