The number of connected smart devices has been increasing exponentially fordifferent Internet-of-Things (IoT) applications. Security has been a long runchallenge in the IoT systems which has many attack vectors, security flaws andvulnerabilities. Securing billions of B connected devices in IoT is a must taskto realize the full potential of IoT applications. Recently, researchers haveproposed many security solutions for IoT. Machine learning has been proposed asone of the emerging solutions for IoT security and Reinforcement learning isgaining more popularity for securing IoT systems. Reinforcement learning,unlike other machine learning techniques, can learn the environment by havingminimum information about the parameters to be learned. It solves theoptimization problem by interacting with the environment adapting theparameters on the fly. In this paper, we present an comprehensive survey ofdifferent types of cyber-attacks against different IoT systems and then wepresent reinforcement learning and deep reinforcement learning based securitysolutions to combat those different types of attacks in different IoT systems.Furthermore, we present the Reinforcement learning for securing CPS systems(i.e., IoT with feedback and control) such as smart grid and smarttransportation system. The recent important attacks and countermeasures usingreinforcement learning B in IoT are also summarized in the form of tables. Withthis paper, readers can have a more thorough understanding of IoT securityattacks and countermeasures using Reinforcement Learning, as well as researchtrends in this area.