Skip to content

Commit 52c7ed8

Browse files
feat: add qr decomposition (#200)
* add qr decomposition * Apply suggestions from code review Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * break down into more libraries * Apply suggestions from code review Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * reuse components to reduce duplication * Apply suggestions from code review Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
1 parent 2e24bb6 commit 52c7ed8

42 files changed

Lines changed: 1203 additions & 252 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,7 @@ Refer to the documentation to quickly integrate and utilize the library's signal
2727
| [Optimization](doc/optimization/README.md) | Gradient Descent |
2828
| [Regularization](doc/regularization/README.md) | L1 (Lasso), L2 (Ridge) |
2929
| [Math](doc/math/README.md) | CORDIC, Quaternion, MatrixNorms, Step Response Metrics |
30-
| [Solvers](doc/solvers/README.md) | Gaussian Elimination, Levinson-Durbin, Durand-Kerner, Cholesky, DARE, Runge-Kutta ODE Integrators (RK4 + Dormand-Prince), Spectral Radius & Discrete Stability Margin |
30+
| [Solvers](doc/solvers/README.md) | Gaussian Elimination, Levinson-Durbin, Durand-Kerner, Cholesky, DARE, Runge-Kutta ODE Integrators (RK4 + Dormand-Prince), Spectral Radius & Discrete Stability Margin, QR Decomposition (Householder / Givens) |
3131
| [Performance Optimization](doc/performance-optimization/README.md) | Compiler optimizations, SIMD |
3232

3333
Each category page lists its algorithms with a brief description and links to the detailed documentation.

‎ROADMAP.md‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,6 @@ Difficulty legend:
2727

2828
| # | Component | Target module | Difficulty |
2929
|----|------------------------------------------------------|---------------------------|------------|
30-
| 27 | QR decomposition (Householder / Givens) | `solvers` | ★★★★☆ |
3130
| 28 | LU decomposition with partial pivoting | `solvers` | ★★★★☆ |
3231
| 29 | Matrix exponential (scaling & squaring + Padé) | `math` | ★★★★☆ |
3332
| 30 | Continuous → discrete conversion (`c2d`) | `math` | ★★★★☆ |

‎doc/math/GivensRotation.md‎

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
# Givens Rotation
2+
3+
## Overview & Motivation
4+
5+
A Givens rotation zeros a single matrix entry by rotating two rows in a plane, leaving all others untouched. Because it touches only two rows, it is the tool of choice for *incremental* linear algebra: streaming a new row into an existing QR factor, sparse triangularization, and the implicit-shift sweeps of QR/SVD eigen-iterations. `ComputeGivens` derives the rotation coefficients; `ApplyGivens` applies them to a scalar pair.
6+
7+
## Mathematical Theory
8+
9+
Given two scalars $a$ (the value to keep) and $b$ (the value to zero), the rotation
10+
11+
$$G = \begin{bmatrix} c & s \\ -s & c \end{bmatrix}, \quad c = \frac{a}{r},\; s = \frac{b}{r},\; r = \sqrt{a^2 + b^2}$$
12+
13+
is orthogonal ($c^2 + s^2 = 1$) and satisfies
14+
15+
$$G \begin{bmatrix} a \\ b \end{bmatrix} = \begin{bmatrix} r \\ 0 \end{bmatrix}$$
16+
17+
Applied across a pair of rows, it zeros the target entry while preserving Euclidean length.
18+
19+
## Complexity Analysis
20+
21+
| Operation | Time | Space | Notes |
22+
|-----------------|--------|--------|----------------------------------|
23+
| ComputeGivens | $O(1)$ | $O(1)$ | One `sqrt`, two divides |
24+
| ApplyGivens | $O(1)$ | $O(1)$ | Rotates one scalar pair |
25+
| Rotate two rows | $O(n)$ | $O(1)$ | `ApplyGivens` across `n` columns |
26+
27+
## Step-by-Step Walkthrough
28+
29+
`ComputeGivens(3, 4)`: $r = 5$, $c = 0.6$, $s = 0.8$. Applying to $(x, y) = (3, 4)$: $x' = 0.6\cdot 3 + 0.8\cdot 4 = 5$, $y' = -0.8\cdot 3 + 0.6\cdot 4 = 0$ — the second component is annihilated and the norm is preserved.
30+
31+
## Pitfalls & Edge Cases
32+
33+
**Degenerate pair** — when $a = b = 0$ the rotation is undefined; `ComputeGivens` returns the identity $(c, s) = (1, 0)$ so applying it is a safe no-op.
34+
35+
**Float-only** — `static_assert(std::is_floating_point_v<T>)`.
36+
37+
## Variants & Generalizations
38+
39+
Householder reflectors (`math::HouseholderTransform`) zero an entire sub-column at once and are cheaper for dense factorization; Givens wins when only one entry (or one streamed row) changes. Fast/"square-root-free" Givens variants trade the `sqrt` for extra bookkeeping.
40+
41+
## Applications
42+
43+
- **Streaming QR update** — rotate a new row into an existing `R` (`QrDecomposition::GivensUpdateRow`).
44+
- **QR / SVD iterations** — implicit-shift bulge chasing.
45+
- **Sparse triangularization** — zero isolated entries without touching the rest.
46+
47+
## Connections to Other Algorithms
48+
49+
Used by `solvers::QrDecomposition`; complements `math::HouseholderTransform`.
50+
51+
## References & Further Reading
52+
53+
- Givens, W., "Computation of Plane Unitary Rotations Transforming a General Matrix to Triangular Form", SIAM J. Appl. Math. 6(1), 1958
54+
- Golub, G. H. & Van Loan, C. F., "Matrix Computations", 4th ed., §5.1 (Givens rotations)

‎doc/math/HouseholderTransform.md‎

Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,55 @@
1+
# Householder Transform
2+
3+
## Overview & Motivation
4+
5+
A Householder reflector is the workhorse of numerically stable dense linear algebra. A single reflector mirrors a vector onto a coordinate axis, zeroing every entry below a chosen pivot in one orthogonal step. Chaining reflectors triangularizes a matrix (QR), bidiagonalizes it (SVD), or tridiagonalizes a symmetric matrix (eigensolvers). `HouseholderVector` computes the reflector for one sub-column; it is the shared primitive those factorizations call.
6+
7+
## Mathematical Theory
8+
9+
For a vector $x$, the Householder reflector is the orthogonal matrix
10+
11+
$$H = I - \beta\, v v^\top$$
12+
13+
chosen so that $Hx$ is zero below the pivot. With $\sigma = \sum_{i>\text{start}} x_i^2$ and $\|x\| = \sqrt{x_{\text{start}}^2 + \sigma}$, the reflector maps $x_{\text{start}} \mapsto \mp\|x\|$. The pivot sign is chosen as $-\operatorname{sign}(x_{\text{start}})\|x\|$ to avoid cancellation:
14+
15+
$$v_{\text{start}} = 1, \quad v_i = x_i / v_0, \quad \beta = \frac{2 v_0^2}{\sigma + v_0^2}$$
16+
17+
$H$ is symmetric and orthogonal ($H = H^\top = H^{-1}$), so applying it is backward stable.
18+
19+
## Complexity Analysis
20+
21+
| Operation | Time | Space | Notes |
22+
|-----------------------|--------|--------|-----------------------------------------|
23+
| HouseholderVector | $O(n)$ | $O(1)$ | Builds `v` (stored implicit unit pivot) |
24+
| Apply $H$ to a vector | $O(n)$ | $O(1)$ | `x - β·v·(vᵀx)` — never form `H` |
25+
26+
## Step-by-Step Walkthrough
27+
28+
For $x = [4, 3, 0, 0]^\top$, pivot `start = 0`: $\sigma = 9$, $\|x\| = 5$. Since $x_0 > 0$, $v_0 = -\sigma/(x_0 + \|x\|) = -1$, giving $\beta = 1$ and $v = [1, -3, 0, 0]^\top$. Reflecting yields $Hx = [-5, 0, 0, 0]^\top$ — the sub-column collapsed onto the axis.
29+
30+
## Pitfalls & Edge Cases
31+
32+
**Already-zero sub-column** — when $\sigma \approx 0$ there is nothing to zero; the routine returns $\beta = 0$ (identity reflector) and callers skip the update.
33+
34+
**Never form `H` explicitly** — apply it as `x − β·v·(vᵀx)` to keep the cost $O(n)$ per column instead of $O(n^2)$.
35+
36+
**Float-only** — `static_assert(std::is_floating_point_v<T>)`; the sign-fixing and normalisation assume real floating-point arithmetic.
37+
38+
## Variants & Generalizations
39+
40+
Givens rotations (`math::GivensRotation`) achieve the same zeroing one entry at a time, preferable for sparse or streaming updates. Complex Householder reflectors extend this to unitary triangularization.
41+
42+
## Applications
43+
44+
- **QR decomposition** — one reflector per column triangularizes `A`.
45+
- **SVD / eigensolvers** — bidiagonalization and tridiagonalization.
46+
- **Square-root Kalman filtering** — covariance factor updates.
47+
48+
## Connections to Other Algorithms
49+
50+
Used by `solvers::QrDecomposition`; complements `math::GivensRotation` and `math::SolveUpperTriangular`. Operates on `math::Vector`.
51+
52+
## References & Further Reading
53+
54+
- Householder, A. S., "Unitary Triangularization of a Nonsymmetric Matrix", JACM 5(4), 1958
55+
- Golub, G. H. & Van Loan, C. F., "Matrix Computations", 4th ed., §5.1 (Householder reflections)

‎doc/math/MatrixNorms.md‎

Lines changed: 7 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -8,13 +8,13 @@ Vector normalisation, closely related, produces the unit-length direction of a v
88

99
## Mathematical Theory
1010

11-
### Vector Norm
11+
### Dot Product & Vector Norm
1212

13-
For a vector $\mathbf{v} \in \mathbb{R}^n$, the Euclidean (L2) norm is
13+
For vectors $\mathbf{a}, \mathbf{b} \in \mathbb{R}^n$, the dot product is $\mathbf{a}\cdot\mathbf{b} = \sum_{i=1}^{n} a_i b_i$. The Euclidean (L2) norm is the square root of the self dot product:
1414

15-
$$\|\mathbf{v}\|_2 = \sqrt{\sum_{i=1}^{n} v_i^2}$$
15+
$$\|\mathbf{v}\|_2 = \sqrt{\mathbf{v}\cdot\mathbf{v}} = \sqrt{\sum_{i=1}^{n} v_i^2}$$
1616

17-
The unit vector $\hat{\mathbf{v}} = \mathbf{v} / \|\mathbf{v}\|_2$ satisfies $\|\hat{\mathbf{v}}\|_2 = 1$. Normalisation is undefined when $\|\mathbf{v}\|_2 = 0$ and must be guarded.
17+
`VectorNorm` is implemented in terms of `DotProduct`. The unit vector $\hat{\mathbf{v}} = \mathbf{v} / \|\mathbf{v}\|_2$ satisfies $\|\hat{\mathbf{v}}\|_2 = 1$. Normalisation is undefined when $\|\mathbf{v}\|_2 = 0$ and must be guarded.
1818

1919
### Matrix Norms
2020

@@ -36,8 +36,9 @@ The 1-norm and infinity-norm are dual: $\|A\|_\infty = \|A^\top\|_1$.
3636

3737
## Complexity Analysis
3838

39-
| Operation | Time | Space | Notes |
40-
|---------------|---------|--------|------------------------------|
39+
| Operation | Time | Space | Notes |
40+
|---------------|---------|--------|----------------------------------------|
41+
| DotProduct | $O(n)$ | $O(1)$ | Single pass; `VectorNorm` builds on it |
4142
| FrobeniusNorm | $O(mn)$ | $O(1)$ | Single pass, no allocation |
4243
| OneNorm | $O(mn)$ | $O(1)$ | Column-wise sum, running max |
4344
| InfinityNorm | $O(mn)$ | $O(1)$ | Row-wise sum, running max |

‎doc/math/MatrixOperations.md‎

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
# Matrix Operations
2+
3+
## Overview & Motivation
4+
5+
Small structural matrix utilities that are needed across the library but do not belong to any single algorithm:
6+
7+
- **`Symmetrize`** enforces exact symmetry on a matrix that should be symmetric in theory but drifts under floating-point round-off.
8+
- **`CongruenceTransform`** computes the quadratic form `A·M·Aᵀ` — the single most common shape in covariance-propagating code (Kalman predict `F·P·Fᵀ`, innovation covariance `H·P·Hᵀ`, Joseph update `(I−KH)·P·(I−KH)ᵀ`, etc.).
9+
10+
Both are recurring needs in Kalman filters, EM parameter updates, and Riccati/Lyapunov solvers.
11+
12+
## Mathematical Theory
13+
14+
Any square matrix decomposes into a symmetric and a skew-symmetric part:
15+
16+
$$M = \underbrace{\tfrac{1}{2}(M + M^\top)}_{\text{symmetric}} + \underbrace{\tfrac{1}{2}(M - M^\top)}_{\text{skew}}$$
17+
18+
`Symmetrize` returns the symmetric part $\tfrac{1}{2}(M + M^\top)$. It is the orthogonal projection (in the Frobenius inner product) of $M$ onto the subspace of symmetric matrices, so it is the *closest* symmetric matrix to $M$. Applying it to an already-symmetric matrix is a no-op (idempotent).
19+
20+
If $M$ is symmetric, `CongruenceTransform` returns a symmetric result exactly (in exact arithmetic): $(AMA^\top)^\top = A M^\top A^\top = A M A^\top$. This makes it the natural building block for propagating a covariance $P$ through a linear map $A$: $P \mapsto A P A^\top$.
21+
22+
## Complexity Analysis
23+
24+
| Operation | Time | Space | Notes |
25+
|---------------------|--------------------|----------|---------------------------------------------------------|
26+
| Symmetrize | $O(n^2)$ | $O(n^2)$ | One transpose, add, scale |
27+
| CongruenceTransform | $O(n^2 m + n m^2)$ | $O(nm)$ | For $A \in \mathbb{R}^{n\times m}$, two matrix products |
28+
29+
## Step-by-Step Walkthrough
30+
31+
**Symmetrize** — for $M = \begin{bmatrix}1 & 3 \\ -1 & 2\end{bmatrix}$: $M^\top = \begin{bmatrix}1 & -1 \\ 3 & 2\end{bmatrix}$, so $\tfrac{1}{2}(M + M^\top) = \begin{bmatrix}1 & 1 \\ 1 & 2\end{bmatrix}$ — off-diagonals averaged, diagonal unchanged.
32+
33+
**CongruenceTransform** — with $A \in \mathbb{R}^{n\times m}$ and symmetric $M \in \mathbb{R}^{m\times m}$, the result $A M A^\top \in \mathbb{R}^{n\times n}$ is the covariance of $A x$ when $x$ has covariance $M$.
34+
35+
## Pitfalls & Edge Cases
36+
37+
**Symmetrize is not a fix for indefiniteness** — it removes the skew part but does not make a matrix positive-definite; covariance code typically also adds a small diagonal jitter (`+ εI`) separately.
38+
39+
**CongruenceTransform association** — the implementation evaluates `(A·M)·Aᵀ`, matching the left-associative `operator*`; results are bit-identical to hand-written `A * M * A.Transpose()`. Under `fast-math` the symmetry of the output can still carry tiny round-off asymmetry — follow with `Symmetrize` when exact symmetry is required.
40+
41+
**Float-only** — both are `static_assert(std::is_floating_point_v<T>)`.
42+
43+
## Variants & Generalizations
44+
45+
The skew-symmetric part $\tfrac{1}{2}(M - M^\top)$ is `Symmetrize`'s companion. The transposed congruence $A^\top M A$ (used by `DiscreteAlgebraicRiccatiEquation`) is obtained by passing `A.Transpose()`.
46+
47+
## Applications
48+
49+
- **Kalman predict / update** — `F·P·Fᵀ`, `H·P·Hᵀ`, Joseph form `(I−KH)·P·(I−KH)ᵀ + K·R·Kᵀ` across the KF/EKF/UKF/smoother family.
50+
- **EM / covariance updates** — re-symmetrize `Q`, `R`, `P` after asymmetric matrix products (`estimators::ExpectationMaximization`).
51+
- **Riccati / Lyapunov solutions** — propagate and enforce symmetry of the solution matrix.
52+
53+
## Connections to Other Algorithms
54+
55+
Operate on `math::Matrix` / `math::SquareMatrix`. Consumed by the `filters::active` Kalman family and `estimators::ExpectationMaximization`; `CongruenceTransform` pairs naturally with `Symmetrize` for covariance-positivity hygiene.
56+
57+
## References & Further Reading
58+
59+
- Golub, G. H. & Van Loan, C. F., "Matrix Computations", 4th ed., §2 (symmetric/skew decomposition)
60+
- Higham, N. J., "Accuracy and Stability of Numerical Algorithms", 2nd ed. (symmetry enforcement in covariance recursions)

‎doc/math/README.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,4 +9,8 @@ Core mathematical primitives for numerical computation.
99
| [CORDIC](Cordic.md) | Iterative shift-add engine for sin/cos, atan2, magnitude, and vector rotation — no multiplier |
1010
| [Quaternion](Quaternion.md) | Unit-quaternion rotation type: Hamilton product, SLERP, rotation-matrix and Euler conversions |
1111
| [MatrixNorms](MatrixNorms.md) | Frobenius, 1-norm, infinity-norm on matrices; vector L2 norm/normalize |
12+
| [Householder Transform](HouseholderTransform.md) | Householder reflector for a sub-column — orthogonal, backward-stable factorization primitive |
13+
| [Givens Rotation](GivensRotation.md) | Plane rotation zeroing one entry — streaming/sparse factorization primitive |
14+
| [Triangular Solve](TriangularSolve.md) | Upper-triangular back-substitution shared by Gaussian elimination and QR |
15+
| [Matrix Operations](MatrixOperations.md) | Structural matrix utilities — `Symmetrize` (closest symmetric matrix) |
1216
| [Step Response Metrics](StepResponseMetrics.md) | Rise time, settling time, percent overshoot, peak time, and steady-state error from a bounded step-response vector |

‎doc/math/TriangularSolve.md‎

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
# Triangular Solve
2+
3+
## Overview & Motivation
4+
5+
Nearly every dense linear solver ends with a triangular system. Gaussian elimination reduces `Ax = b` to an upper-triangular `Ux = c`; QR factorization solves least squares through `Rx = Qᵀb`; Cholesky solves two triangular systems back-to-back. `SolveUpperTriangular` is the shared back-substitution kernel these routines call so the algorithm lives in exactly one place.
6+
7+
## Mathematical Theory
8+
9+
Given an upper-triangular matrix $R \in \mathbb{R}^{n \times n}$ (entries below the diagonal ignored) and a right-hand side $c$, back-substitution solves $Rx = c$ bottom-up:
10+
11+
$$x_i = \frac{1}{r_{ii}}\left(c_i - \sum_{j=i+1}^{n} r_{ij}\, x_j\right), \quad i = n, n-1, \dots, 1$$
12+
13+
Each unknown depends only on those already computed, so a single reverse sweep suffices.
14+
15+
## Complexity Analysis
16+
17+
| Operation | Time | Space | Notes |
18+
|----------------------|----------|--------|---------------------------------|
19+
| SolveUpperTriangular | $O(n^2)$ | $O(n)$ | One reverse sweep, stack output |
20+
21+
## Step-by-Step Walkthrough
22+
23+
For $R = \begin{bmatrix}2 & -1 & 3 \\ 0 & 4 & 1 \\ 0 & 0 & 5\end{bmatrix}$, $c = [10,\, -5,\, 15]^\top$:
24+
25+
1. $x_3 = 15 / 5 = 3$
26+
2. $x_2 = (-5 - 1\cdot 3)/4 = -2$
27+
3. $x_1 = (10 - (-1)(-2) - 3\cdot 3)/2 = 1$ ⇒ $x = [1,\, -2,\, 3]^\top$.
28+
29+
## Pitfalls & Edge Cases
30+
31+
**Singular / near-zero pivot** — a zero diagonal entry makes the system unsolvable. The routine asserts `|r_{ii}| > 0` via `really_assert`; callers (`GaussianElimination`, `QrDecomposition`) detect rank deficiency before reaching it.
32+
33+
**Generic on `T`** — the kernel is templated on any supported numeric type (`float`, `Q15`, `Q31`) because Gaussian elimination is instantiated for all three; it uses `math::ToFloat` only for the pivot assertion.
34+
35+
## Variants & Generalizations
36+
37+
A lower-triangular forward-substitution is the mirror image (top-down sweep) and can be added when Cholesky/LU forward solves need it. Block triangular solves generalise this to matrix right-hand sides.
38+
39+
## Applications
40+
41+
- **Gaussian elimination** — final back-substitution after forward elimination.
42+
- **QR least squares** — solving `Rx = Qᵀb`.
43+
- Any factor-then-solve routine producing a triangular factor.
44+
45+
## Connections to Other Algorithms
46+
47+
Shared by `solvers::GaussianElimination` and `solvers::QrDecomposition`. Operates on `math::Matrix` / `math::Vector`.
48+
49+
## References & Further Reading
50+
51+
- Golub, G. H. & Van Loan, C. F., "Matrix Computations", 4th ed., §3.1 (triangular systems)
52+
- Trefethen, L. N. & Bau, D., "Numerical Linear Algebra", Lecture 17

0 commit comments

Comments
 (0)