Abstract
Reinforcement learning has recently gained unprecedented popularity, yet itstill grapples with sample inefficiency. Addressing this challenge, federatedreinforcement learning (FedRL) has emerged, wherein agents collaborativelylearn a single policy by aggregating local estimations. However, thisaggregation step incurs significant communication costs. In this paper, wepropose CompFedRL, a communication-efficient FedRL approach incorporating both\textit{periodic aggregation} and (direct/error-feedback) compressionmechanisms. Specifically, we consider compressed federated $Q$-learning with agenerative model setup, where a central server learns an optimal $Q$-functionby periodically aggregating compressed $Q$-estimates from local agents. For thefirst time, we characterize the impact of these two mechanisms (which haveremained elusive) by providing a finite-time analysis of our algorithm,demonstrating strong convergence behaviors when utilizing either direct orerror-feedback compression. Our bounds indicate improved solution accuracyconcerning the number of agents and other federated hyperparameters whilesimultaneously reducing communication costs. To corroborate our theory, we alsoconduct in-depth numerical experiments to verify our findings, consideringTop-$K$ and Sparsified-$K$ sparsification operators.