Abstract
The effect of representation has been demonstrated in reinforcement learning,from both theoretical and empirical successes. However, the existingrepresentation learning mainly induced from model learning aspects, misaligningwith our RL tasks. This work introduces Spectral Bellman Representation, anovel framework derived from the Inherent Bellman Error (IBE) condition, whichaligns with the fundamental structure of Bellman updates across a space ofpossible value functions, therefore, directly towards value-based RL. Our keyinsight is the discovery of a fundamental spectral relationship: under thezero-IBE condition, the transformation of a distribution of value functions bythe Bellman operator is intrinsically linked to the feature covariancestructure. This spectral connection yields a new, theoretically-groundedobjective for learning state-action features that inherently capture thisBellman-aligned covariance. Our method requires a simple modification toexisting algorithms. We demonstrate that our learned representations enablestructured exploration, by aligning feature covariance with Bellman dynamics,and improve overall performance, particularly in challenging hard-explorationand long-horizon credit assignment tasks. Our framework naturally extends topowerful multi-step Bellman operators, further broadening its impact. SpectralBellman Representation offers a principled and effective path toward learningmore powerful and structurally sound representations for value-basedreinforcement learning.