Abstract
Existing defence mechanisms have demonstrated significant effectiveness inmitigating rule-based Denial-of-Service (DoS) attacks, leveraging predefinedsignatures and static heuristics to identify and block malicious traffic.However, the emergence of AI-driven techniques presents new challenges to SDNsecurity, potentially compromising the efficacy of existing defence mechanisms.In this paper, we introduce~AdaDoS, an adaptive attack model that disruptnetwork operations while evading detection by existing DoS-based detectorsthrough adversarial reinforcement learning (RL). Specifically, AdaDoS modelsthe problem as a competitive game between an attacker, whose goal is toobstruct network traffic without being detected, and a detector, which aims toidentify malicious traffic. AdaDoS can solve this game by dynamically adjustingits attack strategy based on feedback from the SDN and the detector.Additionally, recognising that attackers typically have less information thandefenders, AdaDoS formulates the DoS-like attack as a partially observed Markovdecision process (POMDP), with the attacker having access only to delayinformation between attacker and victim nodes. We address this challenge with anovel reciprocal learning module, where the student agent, with limitedobservations, enhances its performance by learning from the teacher agent, whohas full observational capabilities in the SDN environment. AdaDoS representsthe first application of RL to develop DoS-like attack sequences, capable ofadaptively evading both machine learning-based and rule-based DoS-like attackdetectors.