Vanilla Bayesian Optimization Performs Great in High Dimensions

Title: Vanilla Bayesian Optimization Performs Great in High Dimensions
Abstract:
In Bayesian optimization (BO), complexity and dimensionality are intrinsically interlinked - the higher the problem dimensionality, the harder it is to optimize. A large collection of algorithms aim to make BO more performant in this setting, commonly by imposing various simplifying assumptions on the objective. In this talk, we hypothesize that the shortcomings of vanilla BO in high dimensions are strictly a consequence of the assumed excessive complexity of the objective. To this end, we view the structural assumptions of existing high-dimensional BO approaches through the lens of model complexity, and modify the assumptions of vanilla BO to be of similarly low complexity. Our enhancement - a simple scaling of the Gaussian process lengthscale prior with the dimensionality - reveals that standard BO works drastically better than previously thought in high dimensions, outperforming state-of-the-art algorithms on tasks with dimensionalities well into the thousands.
Speaker: Carl Hvarfner hvarfner.github.io