Abstract
Federated reinforcement learning (FRL) allows agents to jointly learn aglobal decision-making policy under the guidance of a central server. While FRLhas advantages, its decentralized design makes it prone to poisoning attacks.To mitigate this, Byzantine-robust aggregation techniques tailored for FRL havebeen introduced. Yet, in our work, we reveal that these currentByzantine-robust techniques are not immune to our newly introduced Normalizedattack. Distinct from previous attacks that targeted enlarging the distance ofpolicy updates before and after an attack, our Normalized attack emphasizes onmaximizing the angle of deviation between these updates. To counter thesethreats, we develop an ensemble FRL approach that is provably secure againstboth known and our newly proposed attacks. Our ensemble method involvestraining multiple global policies, where each is learnt by a group of agentsusing any foundational aggregation rule. These well-trained global policiesthen individually predict the action for a specific test state. The ultimateaction is chosen based on a majority vote for discrete action systems or thegeometric median for continuous ones. Our experimental results across differentsettings show that the Normalized attack can greatly disrupt non-ensembleByzantine-robust methods, and our ensemble approach offers substantialresistance against poisoning attacks.