Abstract
Model-based reinforcement learning (MBRL) is recognized with the potential tobe significantly more sample-efficient than model-free RL. How an accuratemodel can be developed automatically and efficiently from raw sensory inputs(such as images), especially for complex environments and tasks, is achallenging problem that hinders the broad application of MBRL in the realworld. In this work, we propose a sensing-aware model-based reinforcementlearning system called SAM-RL. Leveraging the differentiable physics-basedsimulation and rendering, SAM-RL automatically updates the model by comparingrendered images with real raw images and produces the policy efficiently. Withthe sensing-aware learning pipeline, SAM-RL allows a robot to select aninformative viewpoint to monitor the task process. We apply our framework toreal world experiments for accomplishing three manipulation tasks: roboticassembly, tool manipulation, and deformable object manipulation. We demonstratethe effectiveness of SAM-RL via extensive experiments. Videos are available onour project webpage at https://sites.google.com/view/rss-sam-rl.