Abstract
The generalization properties of Gaussian processes depend heavily on thechoice of kernel, and this choice remains a dark art. We present the NeuralKernel Network (NKN), a flexible family of kernels represented by a neuralnetwork. The NKN architecture is based on the composition rules for kernels, sothat each unit of the network corresponds to a valid kernel. It can compactlyapproximate compositional kernel structures such as those used by the AutomaticStatistician (Lloyd et al., 2014), but because the architecture isdifferentiable, it is end-to-end trainable with gradient-based optimization. Weshow that the NKN is universal for the class of stationary kernels. Empiricallywe demonstrate pattern discovery and extrapolation abilities of NKN on severaltasks that depend crucially on identifying the underlying structure, includingtime series and texture extrapolation, as well as Bayesian optimization.