Abstract
Optimizing an expensive-to-query function is a common task in science andengineering, where it is beneficial to keep the number of queries to a minimum.A popular strategy is Bayesian optimization (BO), which leverages probabilisticmodels for this task. Most BO today uses Gaussian processes (GPs), or a fewother surrogate models. However, there is a broad set of Bayesian modelingtechniques that could be used to capture complex systems and reduce the numberof queries in BO. Probabilistic programming languages (PPLs) are modern toolsthat allow for flexible model definition, prior specification, modelcomposition, and automatic inference. In this paper, we develop ProBO, a BOprocedure that uses only standard operations common to most PPLs. This allows auser to drop in a model built with an arbitrary PPL and use it directly in BO.We describe acquisition functions for ProBO, and strategies for efficientlyoptimizing these functions given complex models or costly inference procedures.Using existing PPLs, we implement new models to aid in a few challengingoptimization settings, and demonstrate these on model hyperparameter andarchitecture search tasks.