Reinforcement learning shows great potential to solve complex contact-richrobot manipulation tasks. However, the safety of using RL in the real world isa crucial problem, since unexpected dangerous collisions might happen when theRL policy is imperfect during training or in unseen scenarios. In this paper,we propose a contact-safe reinforcement learning framework for contact-richrobot manipulation, which maintains safety in both the task space and jointspace. When the RL policy causes unexpected collisions between the robot armand the environment, our framework is able to immediately detect the collisionand ensure the contact force to be small. Furthermore, the end-effector isenforced to perform contact-rich tasks compliantly, while keeping robust toexternal disturbances. We train the RL policy in simulation and transfer it tothe real robot. Real world experiments on robot wiping tasks show that ourmethod is able to keep the contact force small both in task space and jointspace even when the policy is under unseen scenario with unexpected collision,while rejecting the disturbances on the main task.