|
| 1 | +# Bayesian Optimization |
| 2 | + |
| 3 | +## Overview & Motivation |
| 4 | + |
| 5 | +Bayesian Optimization (BO) is a gradient-free, sample-efficient strategy for optimizing |
| 6 | +black-box functions that are expensive to evaluate. Unlike gradient-based methods (gradient |
| 7 | +descent, Adam), BO does not require access to derivatives. It is ideal for calibrating |
| 8 | +numerical controllers, tuning hyperparameters, or any optimization task where each function |
| 9 | +evaluation is costly — for example, running a full closed-loop simulation. |
| 10 | + |
| 11 | +BO maintains a probabilistic surrogate model (typically a Gaussian Process) of the objective |
| 12 | +function. At each iteration, it uses an acquisition function to select the next query point |
| 13 | +that balances exploration (uncertain regions) with exploitation (promising regions). After |
| 14 | +evaluating the true objective at that point, the surrogate is updated and the process repeats. |
| 15 | + |
| 16 | +## Mathematical Theory |
| 17 | + |
| 18 | +### Gaussian Process Surrogate |
| 19 | + |
| 20 | +A Gaussian Process (GP) defines a distribution over functions: |
| 21 | + |
| 22 | +$$f \sim \mathcal{GP}(m(\mathbf{x}), k(\mathbf{x}, \mathbf{x}'))$$ |
| 23 | + |
| 24 | +With zero mean $m = 0$ and squared-exponential (RBF) kernel: |
| 25 | + |
| 26 | +$$k(\mathbf{x}, \mathbf{x}') = \sigma_f^2 \exp\!\left(-\frac{\|\mathbf{x} - \mathbf{x}'\|^2}{2\ell^2}\right)$$ |
| 27 | + |
| 28 | +where $\ell$ is the length-scale and $\sigma_f^2$ is the signal variance. |
| 29 | + |
| 30 | +Given $n$ observations $\{(\mathbf{x}_i, y_i)\}$, the GP posterior at a new point $\mathbf{x}_*$ is Gaussian: |
| 31 | + |
| 32 | +$$p(f_* | \mathbf{X}, \mathbf{y}, \mathbf{x}_*) = \mathcal{N}(\mu_*, \sigma_*^2)$$ |
| 33 | + |
| 34 | +$$\mu_* = \mathbf{k}_*^T (\mathbf{K} + \sigma_n^2 \mathbf{I})^{-1} \mathbf{y}$$ |
| 35 | + |
| 36 | +$$\sigma_*^2 = k_{**} - \mathbf{k}_*^T (\mathbf{K} + \sigma_n^2 \mathbf{I})^{-1} \mathbf{k}_*$$ |
| 37 | + |
| 38 | +where: |
| 39 | +- $\mathbf{K}_{ij} = k(\mathbf{x}_i, \mathbf{x}_j)$ is the $n \times n$ kernel matrix |
| 40 | +- $\mathbf{k}_{*,i} = k(\mathbf{x}_*, \mathbf{x}_i)$ is the cross-kernel vector |
| 41 | +- $\sigma_n^2$ is the observation noise variance |
| 42 | + |
| 43 | +### Expected Improvement Acquisition Function |
| 44 | + |
| 45 | +For minimization, the Expected Improvement (EI) at $\mathbf{x}$ given best observed value $f^* = \min_i y_i$ is: |
| 46 | + |
| 47 | +$$\text{EI}(\mathbf{x}) = \mathbb{E}[\max(f^* - f(\mathbf{x}), 0)]$$ |
| 48 | + |
| 49 | +For a GP surrogate this evaluates analytically: |
| 50 | + |
| 51 | +$$\text{EI}(\mathbf{x}) = (f^* - \mu_*)\,\Phi(z) + \sigma_*\,\phi(z), \quad z = \frac{f^* - \mu_*}{\sigma_*}$$ |
| 52 | + |
| 53 | +where $\Phi$ is the standard normal CDF and $\phi$ is the PDF. |
| 54 | + |
| 55 | +### Fixed-Size GP Implementation |
| 56 | + |
| 57 | +To avoid dynamic memory allocation, the kernel matrix is pre-allocated at compile-time with |
| 58 | +maximum capacity $M$. When only $n < M$ observations are present, the |
| 59 | +inactive rows and columns are padded with an identity block, preserving positive-definiteness |
| 60 | +and allowing Gaussian elimination with the full $M \times M$ system. The resulting alpha vector |
| 61 | +$\boldsymbol{\alpha} = (\mathbf{K} + \sigma_n^2 \mathbf{I})^{-1} \mathbf{y}$ has zeros in the |
| 62 | +inactive entries. |
| 63 | + |
| 64 | +## Complexity Analysis |
| 65 | + |
| 66 | +| Case | Time per iteration | Space | Notes | |
| 67 | +|---------|----------------------|---------------|--------------------------------| |
| 68 | +| Best | $O(M^3)$ | $O(M^2 + NP)$ | Dominated by GP solve | |
| 69 | +| Average | $O(M^3 + C \cdot P)$ | $O(M^2 + NP)$ | $C$ candidates, $P$ parameters | |
| 70 | +| Worst | $O(M^3 + C \cdot P)$ | $O(M^2 + NP)$ | Same as average | |
| 71 | + |
| 72 | +Where $M$ = maximum observations, $C$ = candidate count, $P$ = parameter count, $N$ = current number of observations. |
| 73 | + |
| 74 | +All memory is pre-allocated at compile-time on the stack — no dynamic allocation required. |
| 75 | + |
| 76 | +## Step-by-Step Walkthrough |
| 77 | + |
| 78 | +**Example**: minimize $f(x) = (x - 0.5)^2$ on $x \in [0, 1]$. |
| 79 | + |
| 80 | +1. **Initialization**: draw 2 random points, e.g. $x_1 = 0.12$, $x_2 = 0.87$. |
| 81 | +2. **Evaluate**: $y_1 = 0.1444$, $y_2 = 0.1369$. Best so far $f^* = 0.1369$. |
| 82 | +3. **Build GP**: compute $2 \times 2$ kernel matrix $\mathbf{K}$ with hyperparameters |
| 83 | + $\ell=1$, $\sigma_f=1$, $\sigma_n=0.01$. |
| 84 | +4. **Solve**: $\boldsymbol{\alpha} = (\mathbf{K} + \sigma_n^2 \mathbf{I})^{-1} \mathbf{y}$. |
| 85 | +5. **Maximize EI**: sample 100 candidates uniformly, compute $\text{EI}(x_k)$ for each, |
| 86 | + select $x_3 = \arg\max \text{EI}$. |
| 87 | +6. **Evaluate**: $y_3 = f(x_3)$. Update GP. Repeat. |
| 88 | +7. **Convergence** (after 20 iterations): best point converges near $x = 0.5$. |
| 89 | + |
| 90 | +## Pitfalls & Edge Cases |
| 91 | + |
| 92 | +- **GP kernel matrix conditioning**: If two observations are nearly identical, $\mathbf{K} + \sigma_n^2 \mathbf{I}$ |
| 93 | + can be ill-conditioned. The noise variance $\sigma_n^2$ acts as a regularizer; set it $\geq 10^{-3}$ for stability. |
| 94 | +- **Exploration vs exploitation**: A small candidate sample count (below ~30) may miss the EI maximum in high dimensions. |
| 95 | + Increase the candidate count for $P > 3$ parameters. |
| 96 | +- **Bounds scaling**: EI maximization by uniform sampling from $[-1, 1]^P$ assumes a normalized search space. |
| 97 | + Provide appropriate bounds and the optimizer will rescale internally. |
| 98 | +- **Fixed capacity**: The maximum observation budget is a compile-time parameter. Set it large enough for the intended |
| 99 | + number of BO iterations. Exceeding the capacity at runtime triggers a precondition failure. |
| 100 | + |
| 101 | +## Variants & Generalizations |
| 102 | + |
| 103 | +- **Upper Confidence Bound (UCB)**: alternative to EI: $\text{UCB}(\mathbf{x}) = \mu_* - \kappa \sigma_*$. |
| 104 | + More explicit trade-off via $\kappa$. |
| 105 | +- **GP hyperparameter optimization**: Maximize the log marginal likelihood over $\ell$, $\sigma_f$, $\sigma_n$ |
| 106 | + to adapt the surrogate to the observed data automatically. |
| 107 | +- **Gradient-enhanced BO**: When gradients are available, use GP regression on $(f, \nabla f)$ jointly |
| 108 | + for faster convergence. |
| 109 | +- **Batch BO**: Select $q > 1$ candidates per iteration (parallel evaluations) using the $q$-EI criterion. |
| 110 | + |
| 111 | +## Applications |
| 112 | + |
| 113 | +- Controller weight tuning (e.g., MPC state/control cost weights Q, R) |
| 114 | +- Hyperparameter optimisation for machine learning models |
| 115 | +- Experiment design in system identification |
| 116 | +- Any engineering optimisation where each evaluation is costly (simulation, hardware in the loop) |
| 117 | + |
| 118 | +## Connections to Other Algorithms |
| 119 | + |
| 120 | +- **Gaussian Process (GP)**: The surrogate model is a GP; the Kalman filter is formally equivalent to |
| 121 | + GP regression with a Markovian kernel. |
| 122 | +- **Gaussian Elimination / Cholesky**: Used to solve the GP kernel system $(\mathbf{K} + \sigma_n^2 \mathbf{I})\boldsymbol{\alpha} = \mathbf{y}$. |
| 123 | +- **Gradient Descent**: BO is the gradient-free counterpart — preferred when the objective is non-differentiable |
| 124 | + or noisy. |
| 125 | +- **Expectation-Maximisation (EM)**: Both maintain a probabilistic model refined by data; EM operates on |
| 126 | + latent-variable likelihood while BO operates on a surrogate of an arbitrary black-box. |
| 127 | + |
| 128 | +## Numerical Properties |
| 129 | + |
| 130 | +- **Real-time suitability**: No — the $O(M^3)$ GP linear solve at each iteration precludes hard real-time use. |
| 131 | + Suitable for offline calibration or non-real-time configuration stages. |
| 132 | +- **Precision**: Floating-point arithmetic only. Gaussian process kernels require transcendental |
| 133 | + functions — exponential, complementary error function, square root — that are incompatible with |
| 134 | + fixed-point representations. The kernel matrix inversion loses precision for very flat kernels |
| 135 | + ($\ell \gg$ domain width). Length-scale should remain in the range $[0.1, 10]$ relative to the |
| 136 | + normalised parameter domain. |
| 137 | +- **Noise regularisation**: The diagonal noise term $\sigma_n^2 \mathbf{I}$ must be positive to keep the |
| 138 | + kernel matrix positive-definite; values below $10^{-3}$ risk ill-conditioning. |
| 139 | +- **Memory**: All storage is pre-allocated at compile-time — the maximum observation budget and the |
| 140 | + candidate count are both compile-time parameters. No dynamic allocation is required. |
| 141 | + |
| 142 | +## References & Further Reading |
| 143 | + |
| 144 | +- Brochu, Cora, de Freitas, "A Tutorial on Bayesian Optimization of Expensive Cost Functions", 2010 |
| 145 | +- Rasmussen & Williams, *Gaussian Processes for Machine Learning*, MIT Press, 2006 — Chapter 2 (GP regression), Chapter 5 (model selection) |
| 146 | +- Snoek, Larochelle, Adams, "Practical Bayesian Optimization of Machine Learning Algorithms", NeurIPS 2012 |
| 147 | + |
0 commit comments