Conditional Density Estimation (CDE) models deal with estimating conditionaldistributions. The conditions imposed on the distribution are the inputs of themodel. CDE is a challenging task as there is a fundamental trade-off betweenmodel complexity, representational capacity and overfitting. In this work, wepropose to extend the model's input with latent variables and use Gaussianprocesses (GP) to map this augmented input onto samples from the conditionaldistribution. Our Bayesian approach allows for the modeling of small datasets,but we also provide the machinery for it to be applied to big data usingstochastic variational inference. Our approach can be used to model densitieseven in sparse data regions, and allows for sharing learned structure betweenconditions. We illustrate the effectiveness and wide-reaching applicability ofour model on a variety of real-world problems, such as spatio-temporal densityestimation of taxi drop-offs, non-Gaussian noise modeling, and few-shotlearning on omniglot images.