Many popular tourist landmarks are captured in a multitude of online, publicphotos. These photos represent a sparse and unstructured sampling of theplenoptic function for a particular scene. In this paper,we present a newapproach to novel view synthesis under time-varying illumination from suchdata. Our approach builds on the recent multi-plane image (MPI) format forrepresenting local light fields under fixed viewing conditions. We introduce anew DeepMPI representation, motivated by observations on the sparsity structureof the plenoptic function, that allows for real-time synthesis ofphotorealistic views that are continuous in both space and across changes inlighting. Our method can synthesize the same compelling parallax andview-dependent effects as previous MPI methods, while simultaneouslyinterpolating along changes in reflectance and illumination with time. We showhow to learn a model of these effects in an unsupervised way from anunstructured collection of photos without temporal registration, demonstratingsignificant improvements over recent work in neural rendering. More informationcan be found crowdsampling.io.