Skip to content

Commit dbf40cc

Browse files
gabrielfrasantosgithub-actions[bot]Copilot
authored
feat: add bayesian opmization (#138)
* fix instructions * add bayesian optmization * Apply suggestions from code review Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * perf: hoist log2pi as static const in ComputeLogLikelihoodContribution Agent-Logs-Url: https://github.com/embedded-pro/numerical-toolbox-cpp/sessions/b8bc90c4-f2c8-4ef5-ac94-2a563c3e528b Co-authored-by: gabrielfrasantos <21131318+gabrielfrasantos@users.noreply.github.com> --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: gabrielfrasantos <21131318+gabrielfrasantos@users.noreply.github.com>
1 parent 36ff77e commit dbf40cc

31 files changed

Lines changed: 1674 additions & 82 deletions

‎.github/agents/executor.agent.md‎

Lines changed: 8 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -240,10 +240,12 @@ namespace filters::passive
240240
**ALWAYS update documentation for every change:**
241241

242242
- Create or update `doc/{domain}/{AlgorithmName}.md` for every new or modified algorithm
243-
- Use `doc/TEMPLATE.md` as the starting template for new documentation files
244-
- Include: mathematical background, implementation details, usage examples, numerical properties
243+
- Use `doc/TEMPLATE.md` as the starting template — follow its structure exactly
244+
- Documentation is **design-first**: cover mathematical theory, algorithm behaviour, complexity, pitfalls, and connections to other algorithms
245+
- **Do NOT add implementation details** (class names, template parameters, header paths)
246+
- **Do NOT add usage code examples** — the implementation follows the doc, not the other way around
245247
- Update `doc/{domain}/README.md` if a new algorithm is added
246-
- Documentation must stay in sync with code changes
248+
- Keep documentation in sync with algorithmic changes
247249

248250
---
249251

@@ -256,8 +258,9 @@ namespace filters::passive
256258
5. **Create or update tests** using `TYPED_TEST` pattern for every change
257259
6. **Update `CMakeLists.txt`** if new files were added
258260
7. **Update documentation** in `doc/` for every algorithm added or changed
259-
8. **Build and test**: run `cmake --build --preset host` and `ctest --preset host`
260-
9. **Hand off to reviewer** using the handoff button
261+
8. **Add launch configuration** to `.vscode/launch.json` if a new simulator was created — follow the existing `cppdbg` entry pattern; insert before the generic `"Linux Debug"` entry
262+
9. **Build and test**: run `cmake --build --preset host` and `ctest --preset host`
263+
10. **Hand off to reviewer** using the handoff button
261264

262265
## What NOT to Do
263266

‎.github/agents/planner.agent.md‎

Lines changed: 8 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -67,13 +67,15 @@ For each file to create or modify, specify:
6767

6868
#### Documentation Update
6969
- Documentation file to create or update: `doc/{domain}/{AlgorithmName}.md`
70-
- Mathematical background section
71-
- Implementation details and considerations
72-
- Usage examples
70+
- Follow `doc/TEMPLATE.md` structure exactly
71+
- Mathematical background: theory, derivations, complexity, pitfalls, connections
7372
- Numerical properties and limitations (stability, range, precision)
73+
- **No implementation details** (class names, template parameters, header paths)
74+
- **No usage code examples** — docs describe algorithm design; implementation follows from them
7475

7576
#### Build Integration
7677
- `CMakeLists.txt` changes using `numerical_add_header_library()` and `numerical_add_coverage_sources()`
78+
- If a new simulator is created: add a `cppdbg` launch entry to `.vscode/launch.json` following the existing pattern (program path, GDB setup, insert before `"Linux Debug"`)
7779
- Build commands: `cmake --preset host && cmake --build --preset host`
7880
- Test commands: `ctest --preset host`
7981

@@ -148,8 +150,9 @@ Before finalizing, verify the plan against these constraints:
148150

149151
### Documentation — ALWAYS UPDATED
150152
- [ ] `doc/{domain}/{AlgorithmName}.md` created or updated for every change
153+
- [ ] Follows `doc/TEMPLATE.md` structure
151154
- [ ] Mathematical background with equations
152-
- [ ] Implementation details and considerations
153-
- [ ] Usage examples with code snippets
154155
- [ ] Numerical properties: stability, range, precision, real-time suitability
156+
- [ ] No implementation details (class names, template params, header paths)
157+
- [ ] No usage code examples or code snippets
155158
- [ ] README in domain's `doc/{domain}/README.md` updated if needed

‎.github/agents/reviewer.agent.md‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -160,9 +160,10 @@ End with a summary: total criticals, warnings, suggestions, and overall verdict
160160
### 13. Documentation Alignment (CRITICAL)
161161

162162
- [ ] `doc/{domain}/{AlgorithmName}.md` exists and is up-to-date for every changed algorithm
163+
- [ ] Follows `doc/TEMPLATE.md` structure
163164
- [ ] Documentation includes mathematical background (equations, formulas)
164-
- [ ] Documentation includes implementation details and considerations
165-
- [ ] Documentation includes usage examples with code snippets
165+
- [ ] Documentation does **not** include implementation details (class names, template params, header paths)
166+
- [ ] Documentation does **not** include usage code examples or code snippets
166167
- [ ] Documentation includes numerical properties: stability, range, precision, real-time suitability
167168
- [ ] `doc/{domain}/README.md` updated if a new algorithm was added
168169
- [ ] README references correct numeric type support
@@ -176,6 +177,7 @@ End with a summary: total criticals, warnings, suggestions, and overall verdict
176177
- [ ] `target_link_libraries` uses `${NUMERICAL_VISIBILITY}` variable
177178
- [ ] No circular dependencies between targets
178179
- [ ] Test subdirectory added via `add_subdirectory(test)`
180+
- [ ] If a new simulator was created: `.vscode/launch.json` has a new `cppdbg` entry with the correct program path and inserted before the generic `"Linux Debug"` entry
179181

180182
### 15. Code Quality Tools (WARNING)
181183

‎.github/copilot-instructions.md‎

Lines changed: 29 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -73,11 +73,11 @@ Place code here when it is:
7373

7474
When implementing or modifying algorithms:
7575
- **ALWAYS UPDATE DOCUMENTATION**: Each algorithm must have corresponding documentation in `doc/`
76-
- Include mathematical background and theory
77-
- Provide implementation details and considerations
78-
- Add usage examples and expected behaviors
79-
- Document numerical properties and limitations
80-
- Keep documentation in sync with code changes
76+
- Follow the structure defined in `doc/TEMPLATE.md` exactly
77+
- Documentation is **design-first**: describe mathematical theory, algorithm behaviour, complexity, pitfalls, and connections — not how the code is written
78+
- **Do NOT include implementation details** (e.g., class names, template parameters, header paths)
79+
- **Do NOT include usage examples or code snippets** — the implementation must follow what the doc describes, not the other way around
80+
- Keep documentation in sync with algorithmic changes (not code-level changes)
8181

8282
## Design Principles
8383

@@ -114,6 +114,30 @@ These are the most important architectural directives in this codebase. Every ne
114114
- **Naming convention**: Targets follow `numerical.simulator.<domain>.<algorithm>.<layer>` (e.g., `numerical.simulator.controllers.lqr.application`).
115115
- **Library code reuse**: Use `numerical_add_header_library()` and `numerical_add_coverage_sources()` from `cmake/NumericalHeaderLibrary.cmake` for all `numerical/` targets.
116116
- **Compiler options blocks**: Every simulator target repeats the same MSVC/GCC warning suppression — keep this consistent across all targets.
117+
- **Launch configuration**: Every new simulator MUST have a corresponding entry added to `.vscode/launch.json`. Follow the existing pattern exactly:
118+
```json
119+
{
120+
"name": "<SimulatorName> Simulator",
121+
"type": "cppdbg",
122+
"request": "launch",
123+
"program": "${workspaceFolder}/build/host/simulator/<domain>/<SimulatorFolder>/Debug/numerical.simulator.<domain>.<snake_case_name>",
124+
"args": [],
125+
"stopAtEntry": false,
126+
"cwd": "${workspaceFolder}",
127+
"environment": [],
128+
"externalConsole": false,
129+
"MIMode": "gdb",
130+
"miDebuggerPath": "/usr/bin/gdb",
131+
"setupCommands": [
132+
{
133+
"description": "Enable pretty-printing for gdb",
134+
"text": "-enable-pretty-printing",
135+
"ignoreFailures": true
136+
}
137+
]
138+
}
139+
```
140+
Insert the new entry before the generic `"Linux Debug"` configuration, which must remain last.
117141

118142
## Coding Style and Patterns
119143

@@ -402,8 +426,6 @@ When implementing a new algorithm:
402426
4. **Write comprehensive tests** for all numeric types
403427
5. **Document in `doc/`** with:
404428
- Mathematical background
405-
- Implementation details
406-
- Usage examples
407429
- Numerical considerations (stability, range, precision)
408430
6. **Consider SIMD optimizations** where applicable using `math::SingleInstructionMultipleData`
409431

‎.github/instructions/numerical-cpp.instructions.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -74,6 +74,6 @@ Apply `OPTIMIZE_FOR_SPEED` (from `numerical/math/CompilerOptimizations.hpp`) on
7474

7575
## Documentation — MANDATORY
7676

77-
For every algorithm added or modified, update the corresponding `doc/{domain}/{AlgorithmName}.md` file. Include mathematical background, implementation details, usage examples, and numerical properties.
77+
For every algorithm added or modified, update the corresponding `doc/{domain}/{AlgorithmName}.md` file. Follow `doc/TEMPLATE.md` exactly. Documentation is **design-first**: cover mathematical background, algorithm behaviour, complexity, pitfalls, and connections. Do **not** include implementation details, class names, template parameters, or usage code examples — docs describe the algorithm design; code follows from it.
7878

7979
Full details: [copilot-instructions.md](../../.github/copilot-instructions.md)

‎.vscode/launch.json‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -221,6 +221,26 @@
221221
}
222222
]
223223
},
224+
{
225+
"name": "Bayesian MPC Calibration Simulator",
226+
"type": "cppdbg",
227+
"request": "launch",
228+
"program": "${workspaceFolder}/build/host/simulator/controllers/BayesianMpcCalibration/Debug/numerical.simulator.controllers.bayesian_mpc_calibration",
229+
"args": [],
230+
"stopAtEntry": false,
231+
"cwd": "${workspaceFolder}",
232+
"environment": [],
233+
"externalConsole": false,
234+
"MIMode": "gdb",
235+
"miDebuggerPath": "/usr/bin/gdb",
236+
"setupCommands": [
237+
{
238+
"description": "Enable pretty-printing for gdb",
239+
"text": "-enable-pretty-printing",
240+
"ignoreFailures": true
241+
}
242+
]
243+
},
224244
{
225245
"name": "Linux Debug",
226246
"type": "cppdbg",
Lines changed: 147 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,147 @@
1+
# Bayesian Optimization
2+
3+
## Overview & Motivation
4+
5+
Bayesian Optimization (BO) is a gradient-free, sample-efficient strategy for optimizing
6+
black-box functions that are expensive to evaluate. Unlike gradient-based methods (gradient
7+
descent, Adam), BO does not require access to derivatives. It is ideal for calibrating
8+
numerical controllers, tuning hyperparameters, or any optimization task where each function
9+
evaluation is costly — for example, running a full closed-loop simulation.
10+
11+
BO maintains a probabilistic surrogate model (typically a Gaussian Process) of the objective
12+
function. At each iteration, it uses an acquisition function to select the next query point
13+
that balances exploration (uncertain regions) with exploitation (promising regions). After
14+
evaluating the true objective at that point, the surrogate is updated and the process repeats.
15+
16+
## Mathematical Theory
17+
18+
### Gaussian Process Surrogate
19+
20+
A Gaussian Process (GP) defines a distribution over functions:
21+
22+
$$f \sim \mathcal{GP}(m(\mathbf{x}), k(\mathbf{x}, \mathbf{x}'))$$
23+
24+
With zero mean $m = 0$ and squared-exponential (RBF) kernel:
25+
26+
$$k(\mathbf{x}, \mathbf{x}') = \sigma_f^2 \exp\!\left(-\frac{\|\mathbf{x} - \mathbf{x}'\|^2}{2\ell^2}\right)$$
27+
28+
where $\ell$ is the length-scale and $\sigma_f^2$ is the signal variance.
29+
30+
Given $n$ observations $\{(\mathbf{x}_i, y_i)\}$, the GP posterior at a new point $\mathbf{x}_*$ is Gaussian:
31+
32+
$$p(f_* | \mathbf{X}, \mathbf{y}, \mathbf{x}_*) = \mathcal{N}(\mu_*, \sigma_*^2)$$
33+
34+
$$\mu_* = \mathbf{k}_*^T (\mathbf{K} + \sigma_n^2 \mathbf{I})^{-1} \mathbf{y}$$
35+
36+
$$\sigma_*^2 = k_{**} - \mathbf{k}_*^T (\mathbf{K} + \sigma_n^2 \mathbf{I})^{-1} \mathbf{k}_*$$
37+
38+
where:
39+
- $\mathbf{K}_{ij} = k(\mathbf{x}_i, \mathbf{x}_j)$ is the $n \times n$ kernel matrix
40+
- $\mathbf{k}_{*,i} = k(\mathbf{x}_*, \mathbf{x}_i)$ is the cross-kernel vector
41+
- $\sigma_n^2$ is the observation noise variance
42+
43+
### Expected Improvement Acquisition Function
44+
45+
For minimization, the Expected Improvement (EI) at $\mathbf{x}$ given best observed value $f^* = \min_i y_i$ is:
46+
47+
$$\text{EI}(\mathbf{x}) = \mathbb{E}[\max(f^* - f(\mathbf{x}), 0)]$$
48+
49+
For a GP surrogate this evaluates analytically:
50+
51+
$$\text{EI}(\mathbf{x}) = (f^* - \mu_*)\,\Phi(z) + \sigma_*\,\phi(z), \quad z = \frac{f^* - \mu_*}{\sigma_*}$$
52+
53+
where $\Phi$ is the standard normal CDF and $\phi$ is the PDF.
54+
55+
### Fixed-Size GP Implementation
56+
57+
To avoid dynamic memory allocation, the kernel matrix is pre-allocated at compile-time with
58+
maximum capacity $M$. When only $n < M$ observations are present, the
59+
inactive rows and columns are padded with an identity block, preserving positive-definiteness
60+
and allowing Gaussian elimination with the full $M \times M$ system. The resulting alpha vector
61+
$\boldsymbol{\alpha} = (\mathbf{K} + \sigma_n^2 \mathbf{I})^{-1} \mathbf{y}$ has zeros in the
62+
inactive entries.
63+
64+
## Complexity Analysis
65+
66+
| Case | Time per iteration | Space | Notes |
67+
|---------|----------------------|---------------|--------------------------------|
68+
| Best | $O(M^3)$ | $O(M^2 + NP)$ | Dominated by GP solve |
69+
| Average | $O(M^3 + C \cdot P)$ | $O(M^2 + NP)$ | $C$ candidates, $P$ parameters |
70+
| Worst | $O(M^3 + C \cdot P)$ | $O(M^2 + NP)$ | Same as average |
71+
72+
Where $M$ = maximum observations, $C$ = candidate count, $P$ = parameter count, $N$ = current number of observations.
73+
74+
All memory is pre-allocated at compile-time on the stack — no dynamic allocation required.
75+
76+
## Step-by-Step Walkthrough
77+
78+
**Example**: minimize $f(x) = (x - 0.5)^2$ on $x \in [0, 1]$.
79+
80+
1. **Initialization**: draw 2 random points, e.g. $x_1 = 0.12$, $x_2 = 0.87$.
81+
2. **Evaluate**: $y_1 = 0.1444$, $y_2 = 0.1369$. Best so far $f^* = 0.1369$.
82+
3. **Build GP**: compute $2 \times 2$ kernel matrix $\mathbf{K}$ with hyperparameters
83+
$\ell=1$, $\sigma_f=1$, $\sigma_n=0.01$.
84+
4. **Solve**: $\boldsymbol{\alpha} = (\mathbf{K} + \sigma_n^2 \mathbf{I})^{-1} \mathbf{y}$.
85+
5. **Maximize EI**: sample 100 candidates uniformly, compute $\text{EI}(x_k)$ for each,
86+
select $x_3 = \arg\max \text{EI}$.
87+
6. **Evaluate**: $y_3 = f(x_3)$. Update GP. Repeat.
88+
7. **Convergence** (after 20 iterations): best point converges near $x = 0.5$.
89+
90+
## Pitfalls & Edge Cases
91+
92+
- **GP kernel matrix conditioning**: If two observations are nearly identical, $\mathbf{K} + \sigma_n^2 \mathbf{I}$
93+
can be ill-conditioned. The noise variance $\sigma_n^2$ acts as a regularizer; set it $\geq 10^{-3}$ for stability.
94+
- **Exploration vs exploitation**: A small candidate sample count (below ~30) may miss the EI maximum in high dimensions.
95+
Increase the candidate count for $P > 3$ parameters.
96+
- **Bounds scaling**: EI maximization by uniform sampling from $[-1, 1]^P$ assumes a normalized search space.
97+
Provide appropriate bounds and the optimizer will rescale internally.
98+
- **Fixed capacity**: The maximum observation budget is a compile-time parameter. Set it large enough for the intended
99+
number of BO iterations. Exceeding the capacity at runtime triggers a precondition failure.
100+
101+
## Variants & Generalizations
102+
103+
- **Upper Confidence Bound (UCB)**: alternative to EI: $\text{UCB}(\mathbf{x}) = \mu_* - \kappa \sigma_*$.
104+
More explicit trade-off via $\kappa$.
105+
- **GP hyperparameter optimization**: Maximize the log marginal likelihood over $\ell$, $\sigma_f$, $\sigma_n$
106+
to adapt the surrogate to the observed data automatically.
107+
- **Gradient-enhanced BO**: When gradients are available, use GP regression on $(f, \nabla f)$ jointly
108+
for faster convergence.
109+
- **Batch BO**: Select $q > 1$ candidates per iteration (parallel evaluations) using the $q$-EI criterion.
110+
111+
## Applications
112+
113+
- Controller weight tuning (e.g., MPC state/control cost weights Q, R)
114+
- Hyperparameter optimisation for machine learning models
115+
- Experiment design in system identification
116+
- Any engineering optimisation where each evaluation is costly (simulation, hardware in the loop)
117+
118+
## Connections to Other Algorithms
119+
120+
- **Gaussian Process (GP)**: The surrogate model is a GP; the Kalman filter is formally equivalent to
121+
GP regression with a Markovian kernel.
122+
- **Gaussian Elimination / Cholesky**: Used to solve the GP kernel system $(\mathbf{K} + \sigma_n^2 \mathbf{I})\boldsymbol{\alpha} = \mathbf{y}$.
123+
- **Gradient Descent**: BO is the gradient-free counterpart — preferred when the objective is non-differentiable
124+
or noisy.
125+
- **Expectation-Maximisation (EM)**: Both maintain a probabilistic model refined by data; EM operates on
126+
latent-variable likelihood while BO operates on a surrogate of an arbitrary black-box.
127+
128+
## Numerical Properties
129+
130+
- **Real-time suitability**: No — the $O(M^3)$ GP linear solve at each iteration precludes hard real-time use.
131+
Suitable for offline calibration or non-real-time configuration stages.
132+
- **Precision**: Floating-point arithmetic only. Gaussian process kernels require transcendental
133+
functions — exponential, complementary error function, square root — that are incompatible with
134+
fixed-point representations. The kernel matrix inversion loses precision for very flat kernels
135+
($\ell \gg$ domain width). Length-scale should remain in the range $[0.1, 10]$ relative to the
136+
normalised parameter domain.
137+
- **Noise regularisation**: The diagonal noise term $\sigma_n^2 \mathbf{I}$ must be positive to keep the
138+
kernel matrix positive-definite; values below $10^{-3}$ risk ill-conditioning.
139+
- **Memory**: All storage is pre-allocated at compile-time — the maximum observation budget and the
140+
candidate count are both compile-time parameters. No dynamic allocation is required.
141+
142+
## References & Further Reading
143+
144+
- Brochu, Cora, de Freitas, "A Tutorial on Bayesian Optimization of Expensive Cost Functions", 2010
145+
- Rasmussen & Williams, *Gaussian Processes for Machine Learning*, MIT Press, 2006 — Chapter 2 (GP regression), Chapter 5 (model selection)
146+
- Snoek, Larochelle, Adams, "Practical Bayesian Optimization of Machine Learning Algorithms", NeurIPS 2012
147+

‎doc/optimization/README.md‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@ General-purpose optimization algorithms for parameter fitting, training, and con
44

55
## Algorithms
66

7-
| Algorithm | Description |
8-
|----------------------------------|----------------------------------------------------------------|
9-
| [Gradient Descent](Optimizer.md) | Iterative batch optimization via gradient-based weight updates |
7+
| Algorithm | Description |
8+
|--------------------------------------------------|---------------------------------------------------------------------------------------------|
9+
| [Bayesian Optimization](BayesianOptimization.md) | Gradient-free global optimization using Gaussian Process surrogate and Expected Improvement |
10+
| [Gradient Descent](Optimizer.md) | Iterative batch optimization via gradient-based weight updates |

0 commit comments

Comments
 (0)