Learning contact-rich, robotic manipulation skills is a challenging problemdue to the high-dimensionality of the state and action space as well asuncertainty from noisy sensors and inaccurate motor control. To combat thesefactors and achieve more robust manipulation, humans actively exploit contactconstraints in the environment. By adopting a similar strategy, robots can alsoachieve more robust manipulation. In this paper, we enable a robot toautonomously modify its environment and thereby discover how to easemanipulation skill learning. Specifically, we provide the robot with fixturesthat it can freely place within the environment. These fixtures provide hardconstraints that limit the outcome of robot actions. Thereby, they funneluncertainty from perception and motor control and scaffold manipulation skilllearning. We propose a learning system that consists of two learning loops. Inthe outer loop, the robot positions the fixture in the workspace. In the innerloop, the robot learns a manipulation skill and after a fixed number ofepisodes, returns the reward to the outer loop. Thereby, the robot isincentivised to place the fixture such that the inner loop quickly achieves ahigh reward. We demonstrate our framework both in simulation and in the realworld on three tasks: peg insertion, wrench manipulation and shallow-depthinsertion. We show that manipulation skill learning is dramatically sped upthrough this way of scaffolding.