The success of Reinforcement Learning (RL) heavily relies on the ability tolearn robust representations from the observations of the environment. In mostcases, the representations learned purely by the reinforcement learning losscan differ vastly across states depending on how the value functions change.However, the representations learned need not be very specific to the task athand. Relying only on the RL objective may yield representations that varygreatly across successive time steps. In addition, since the RL loss has achanging target, the representations learned would depend on how good thecurrent values/policies are. Thus, disentangling the representations from themain task would allow them to focus not only on the task-specific features butalso the environment dynamics. To this end, we propose locally constrainedrepresentations, where an auxiliary loss forces the state representations to bepredictable by the representations of the neighboring states. This encouragesthe representations to be driven not only by the value/policy learning but alsoby an additional loss that constrains the representations from over-fitting tothe value loss. We evaluate the proposed method on several known benchmarks andobserve strong performance. Especially in continuous control tasks, ourexperiments show a significant performance improvement.