Reinforcement learning algorithms usually assume that all actions are alwaysavailable to an agent. However, both people and animals understand the generallink between the features of their environment and the actions that arefeasible. Gibson (1977) coined the term "affordances" to describe the fact thatcertain states enable an agent to do certain actions, in the context ofembodied agents. In this paper, we develop a theory of affordances for agentswho learn and plan in Markov Decision Processes. Affordances play a dual rolein this case. On one hand, they allow faster planning, by reducing the numberof actions available in any given situation. On the other hand, they facilitatemore efficient and precise learning of transition models from data, especiallywhen such models require function approximation. We establish these propertiesthrough theoretical results as well as illustrative examples. We also proposean approach to learn affordances and use it to estimate transition models thatare simpler and generalize better.