We present a detail-driven deep neural network for point set upsampling. Ahigh-resolution point set is essential for point-based rendering and surfacereconstruction. Inspired by the recent success of neural image super-resolutiontechniques, we progressively train a cascade of patch-based upsampling networkson different levels of detail end-to-end. We propose a series of architecturaldesign contributions that lead to a substantial performance boost. The effectof each technical contribution is demonstrated in an ablation study.Qualitative and quantitative experiments show that our method significantlyoutperforms the state-of-the-art learning-based and optimazation-basedapproaches, both in terms of handling low-resolution inputs and revealinghigh-fidelity details.