We introduce Act2Vec, a general framework for learning context-based actionrepresentation for Reinforcement Learning. Representing actions in a vectorspace help reinforcement learning algorithms achieve better performance bygrouping similar actions and utilizing relations between different actions. Weshow how prior knowledge of an environment can be extracted from demonstrationsand injected into action vector representations that encode natural compatiblebehavior. We then use these for augmenting state representations as well asimproving function approximation of Q-values. We visualize and test actionembeddings in three domains including a drawing task, a high dimensionalnavigation task, and the large action space domain of StarCraft II.