We study the problem of robotic stacking with objects of complex geometry. Wepropose a challenging and diverse set of such objects that was carefullydesigned to require strategies beyond a simple "pick-and-place" solution. Ourmethod is a reinforcement learning (RL) approach combined with vision-basedinteractive policy distillation and simulation-to-reality transfer. Our learnedpolicies can efficiently handle multiple object combinations in the real worldand exhibit a large variety of stacking skills. In a large experimental study,we investigate what choices matter for learning such general vision-basedagents in simulation, and what affects optimal transfer to the real robot. Wethen leverage data collected by such policies and improve upon them withoffline RL. A video and a blog post of our work are provided as supplementarymaterial.