With the help of Deep Neural Networks, Deep Reinforcement Learning (DRL) hasachieved great success on many complex tasks during the past few years. SpikingNeural Networks (SNNs) have been used for the implementation of Deep NeuralNetworks with superb energy efficiency on dedicated neuromorphic hardware, andrecent years have witnessed increasing attention on combining SNNs withReinforcement Learning, whereas most approaches still work with huge energyconsumption and high latency. This work proposes the Adaptive Coding SpikingFramework (ACSF) for SNN-based DRL and achieves low latency and great energyefficiency at the same time. Inspired by classical conditioning in biology, wesimulate receptors, central interneurons, and effectors with spike encoders,SNNs, and spike decoders, respectively. We use our proposed ACSF to estimatethe value function in reinforcement learning and conduct extensive experimentsto verify the effectiveness of our proposed framework.