Abstract
Value-based algorithms are a cornerstone of off-policy reinforcement learningdue to their simplicity and training stability. However, their use hastraditionally been restricted to discrete action spaces, as they rely onestimating Q-values for individual state-action pairs. In continuous actionspaces, evaluating the Q-value over the entire action space becomescomputationally infeasible. To address this, actor-critic methods are typicallyemployed, where a critic is trained on off-policy data to estimate Q-values,and an actor is trained to maximize the critic's output. Despite theirpopularity, these methods often suffer from instability during training. Inthis work, we propose a purely value-based framework for continuous controlthat revisits structural maximization of Q-functions, introducing a set of keyarchitectural and algorithmic choices to enable efficient and stable learning.We evaluate the proposed actor-free Q-learning approach on a range of standardsimulation tasks, demonstrating performance and sample efficiency on par withstate-of-the-art baselines, without the cost of learning a separate actor.Particularly, in environments with constrained action spaces, where the valuefunctions are typically non-smooth, our method with structural maximizationoutperforms traditional actor-critic methods with gradient-based maximization.We have released our code at https://github.com/USC-Lira/Q3C.