Abstract
Policies trained in simulation often fail when transferred to the real worlddue to the `reality gap' where the simulator is unable to accurately capturethe dynamics and visual properties of the real world. Current approaches totackle this problem, such as domain randomization, require prior knowledge andengineering to determine how much to randomize system parameters in order tolearn a policy that is robust to sim-to-real transfer while also not being tooconservative. We propose a method for automatically tuning simulator systemparameters to match the real world using only raw RGB images of the real worldwithout the need to define rewards or estimate state. Our key insight is toreframe the auto-tuning of parameters as a search problem where we iterativelyshift the simulation system parameters to approach the real-world systemparameters. We propose a Search Param Model (SPM) that, given a sequence ofobservations and actions and a set of system parameters, predicts whether thegiven parameters are higher or lower than the true parameters used to generatethe observations. We evaluate our method on multiple robotic control tasks inboth sim-to-sim and sim-to-real transfer, demonstrating significant improvementover naive domain randomization. Project videos and code athttps://yuqingd.github.io/autotuned-sim2real/