Constrained and High-dimensional Bayesian Optimization with Transformers

Abstract

This thesis advances Bayesian Optimization (BO) methodology through two novel algorithms that address critical limitations in handling constraints and high-dimensional spaces. First, we introduce a constraint-handling framework leveraging Prior-data Fitted Networks (PFNs), a foundation transformer model that evaluates objectives and constraints simultaneously in a single forward pass through in-context learning. This approach demonstrates an order of magnitude speedup while maintaining or improving solution quality across 15 test problems spanning synthetic, structural, and engineering design challenges. Second, we propose Gradient-Informed Bayesian Optimization using Tabular Foundation Models (GITBO), which utilizes pre-trained tabular foundation models as surrogates for high-dimensional optimization (exceeding 100 dimensions). By exploiting internal gradient computations to identify sensitive optimization directions, GIT-BO creates continuously re-estimated active subspaces without model retraining. Empirical evaluation across 23 benchmarks demonstrates GIT-BO’s superior performance compared to state-of-the-art Gaussian Process-based methods, particularly as dimensionality increases to 500 dimensions. Together, these approaches establish foundation models as powerful alternatives to Gaussian Process methods for constrained and high-dimensional Bayesian optimization challenges.