Abstract
Despite recent rapid progress in AI safety, current large language modelsremain vulnerable to adversarial attacks in multi-turn interaction settings,where attackers strategically adapt their prompts across conversation turns andpose a more critical yet realistic challenge. Existing approaches that discoversafety vulnerabilities either rely on manual red-teaming with human experts oremploy automated methods using pre-defined templates and human-curated attackdata, with most focusing on single-turn attacks. However, these methods did notexplore the vast space of possible multi-turn attacks, failing to considernovel attack trajectories that emerge from complex dialogue dynamics andstrategic conversation planning. This gap is particularly critical given recentfindings that LLMs exhibit significantly higher vulnerability to multi-turnattacks compared to single-turn attacks. We propose DialTree-RPO, an on-policyreinforcement learning framework integrated with tree search that autonomouslydiscovers diverse multi-turn attack strategies by treating the dialogue as asequential decision-making problem, enabling systematic exploration withoutmanually curated data. Through extensive experiments, our approach not onlyachieves more than 25.9% higher ASR across 10 target models compared toprevious state-of-the-art approaches, but also effectively uncovers new attackstrategies by learning optimal dialogue policies that maximize attack successacross multiple turns.