Abstract
We propose VRL3, a powerful data-driven framework with a simple design forsolving challenging visual deep reinforcement learning (DRL) tasks. We analyzea number of major obstacles in taking a data-driven approach, and present asuite of design principles, novel findings, and critical insights aboutdata-driven visual DRL. Our framework has three stages: in stage 1, we leveragenon-RL datasets (e.g. ImageNet) to learn task-agnostic visual representations;in stage 2, we use offline RL data (e.g. a limited number of expertdemonstrations) to convert the task-agnostic representations into more powerfultask-specific representations; in stage 3, we fine-tune the agent with onlineRL. On a set of challenging hand manipulation tasks with sparse reward andrealistic visual inputs, compared to the previous SOTA, VRL3 achieves anaverage of 780% better sample efficiency. And on the hardest task, VRL3 is1220% more sample efficient (2440% when using a wider encoder) and solves thetask with only 10% of the computation. These significant results clearlydemonstrate the great potential of data-driven deep reinforcement learning.