eve: document the module compression measurements

eve-hwe_defconfig inherits Ubuntu's `# CONFIG_MODULE_COMPRESS_ALL is not
set`, so hwe ships 4429 uncompressed modules while declaring a zstd
algorithm it never uses. That reads as an oversight; it is not, and the
obvious one-line fix makes it worse.

Measured on 6.18.35, rootfs being xz squashfs with the x86 BCJ filter:

  core  xz    on  (committed)   291.97 MiB   --
  core  xz    off               292.79 MiB   +0.82
  hwe   zstd  off (committed)   417.17 MiB   --
  hwe   zstd  on                474.67 MiB   +57.50
  hwe   xz    on                418.44 MiB   +1.27

Per-module compression is near-free only when it matches squashfs's own
algorithm. zstd output is high-entropy, so squashfs can neither dedup nor
recompress it, and its tree is bigger than xz's besides - it loses on both
axes. Leaving modules raw lets squashfs-xz dedup across the whole tree,
which is why uncompressed gives hwe the smallest rootfs.

So setting CONFIG_MODULE_COMPRESS_ALL=y alone lands on the worst row:
+57.50 MiB on every device. Both flavours stay as they are.

Author: rucoder

eve/docs/module-compression.md

@@ -0,0 +1,166 @@
+# Kernel module compression in EVE
+
+**Decision: leave it exactly as it is.** `core` and `rt` compress modules with xz; `hwe` and
+`ai` do not compress them at all. Both are correct, and this document exists so nobody
+"fixes" the second one.
+
+Measured 2026-08-25 on 6.18.35.
+
+---
+
+## Why this document exists
+
+`arch/x86/configs/eve-hwe_defconfig` contains, inherited from Ubuntu's config:
+
+```
+CONFIG_MODULE_COMPRESS=y
+CONFIG_MODULE_COMPRESS_ZSTD=y
+# CONFIG_MODULE_COMPRESS_ALL is not set
+```
+
+`MODULE_COMPRESS_ALL` is `default y` (`kernel/module/Kconfig:384`) and is the switch that makes
+`modules_install` actually compress. Ubuntu turns it off explicitly, so `hwe` ships 4429
+uncompressed `.ko` files — 634.8 MiB — while declaring a compression algorithm it never uses.
+
+That reads like an oversight. It is not, and the obvious one-line fix makes things worse.
+Three kernel builds and three EVE rootfs builds were spent establishing this; the numbers are
+below so it does not have to happen again.
+
+## Results
+
+EVE rootfs is squashfs with **xz**, the x86 BCJ filter and 128 KiB blocks. All rootfs figures
+are `installer/rootfs-generic.img` from `make HV=kvm PLATFORM=generic pkgs rootfs`.
+
+| flavour | algorithm | `MODULE_COMPRESS_ALL` | modules | module tree in image | rootfs.img | Δ rootfs |
+|---|---|---|---:|---:|---:|---:|
+| **core** | xz | **on** — committed | 297 | 16.4 MiB | **291.97 MiB** | — |
+| core | xz | off | 297 | 91.4 MiB | 292.79 MiB | +0.82 MiB |
+| **hwe** | zstd | **off** — committed | 4429 | 634.8 MiB | **417.17 MiB** | — |
+| hwe | zstd | on | 4429 | 196.7 MiB | 474.67 MiB | **+57.50 MiB** |
+| hwe | xz | on | 4429 | 143.7 MiB | 418.44 MiB | +1.27 MiB |
+
+## The mechanism
+
+Per-module compression is nearly free on the rootfs **only when it uses the same algorithm as
+squashfs**. EVE's squashfs is xz, so:
+
+- **xz modules** cost +0.82 MiB (core) and +1.27 MiB (hwe). Squashfs loses its dedup and
+  cross-file window over that data, but per-file xz with a larger dictionary recovers almost
+  all of it.
+- **zstd modules** cost **+57.50 MiB**. The output is high-entropy, so squashfs can neither
+  dedup nor recompress it, and zstd's own ratio is worse than what squashfs-xz achieves over
+  the raw tree. It loses on both axes at once — its module tree (196.7 MiB) is also bigger
+  than xz's (143.7 MiB).
+
+Leaving modules **uncompressed** lets squashfs-xz work across the whole 634.8 MiB tree with
+dedup and a shared dictionary, which is why it produces the smallest rootfs of the three.
+
+## The trap
+
+The tempting change is one line in `common.fragment`:
+
+```
+CONFIG_MODULE_COMPRESS_ALL=y      # DO NOT
+```
+
+Because `eve-hwe_defconfig` selects zstd, that lands on the **worst row in the table**:
++57.50 MiB on every device's rootfs. If you want compression on `hwe` you must also switch the
+algorithm to xz — and even then it costs +1.27 MiB of rootfs and roughly 10-18 minutes of
+build time, to save container-image size.
+
+## Why `hwe` stays uncompressed
+
+Uncompressed wins on the two things that affect deployed devices:
+
+- **smallest rootfs** — 417.17 MiB, better than either compressed variant
+- **fastest module load** — one decompression pass instead of two
+
+It loses only on **container image size**: 634.8 MiB of modules versus 143.7 MiB with xz, so
+roughly 491 MiB more per image pulled, cached and stored. That is a CI and registry cost, not
+a device cost, and EVE optimises for the device.
+
+## Why `core` stays compressed
+
+`core` serves eve-kvm under a hard rootfs cap (`ROOTFS_MAXSIZE_MB`, 290 historically and 295
+after the resize work), and xz compression makes its rootfs *smaller*, not larger. It is
+already `default y` because `eve-core_defconfig` does not carry Ubuntu's opt-out. Nothing to
+do.
+
+For reference, `core` on 6.18.35 measures 291.97 MiB against a shipping 6.12 kvm rootfs of
+290.13 MiB — it clears the 295 cap and misses the 290 one.
+
+## Costs, measured
+
+**Build time** (kernel + EVE rootfs, same machine):
+
+| | |
+|---|---|
+| hwe, no compression | 33 min *(ccache cold)* |
+| hwe, zstd | 43 min *(ccache warm)* |
+| hwe, xz | 51 min *(ccache warm)* |
+
+Compression is added work at `modules_install` and it is not free at 4429 modules.
+
+**Module load** — decompressing all 297 `core` modules, 16.4 MB → 91.4 MB, took 3.75 s
+single-threaded including per-file process spawn (an upper bound; in-kernel decompression
+avoids that). The two passes are not equal in size:
+
+| | squashfs pass | module pass |
+|---|---|---|
+| uncompressed | decompresses 91.4 MB of output | — |
+| compressed | decompresses 16.4 MB of output | 16.4 → 91.4 MB |
+
+Squashfs has 5.6× less to do when modules arrive pre-compressed, so the module pass largely
+*replaces* work rather than adding to it — the net penalty is about **+16%**, not +100%. And
+a real device loads a few dozen modules, not all of them.
+
+## Reproducing this
+
+No push is needed; linuxkit resolves a locally built kernel from `~/.linuxkit/cache`.
+
+```sh
+# build a kernel flavour
+cd eve-kernel
+make -f Makefile.eve KERNEL_CONFIG_FLAVOR=<core|hwe> kernel-gcc
+
+# build an EVE rootfs against it
+cd ../eve
+make KERNEL_TAG=$(make -C ../eve-kernel -s -f Makefile.eve \
+       KERNEL_CONFIG_FLAVOR=<core|hwe> docker-tag-gcc) \
+     HV=kvm PLATFORM=generic pkgs rootfs
+```
+
+Two things that will otherwise waste your time:
+
+- **`pkgs` before `rootfs` is required** if your eve checkout has local modifications. Packages
+  are addressed by source hash, so a modified `pkg/pillar` hashes to a tag that exists in no
+  registry and the pull fails.
+- **The rootfs size check runs *after* the image is written** (`eve/Makefile:935`), so a build
+  that exceeds `ROOTFS_MAXSIZE_MB` still leaves a measurable `rootfs-generic.img`. You do not
+  need to raise the cap to measure something over it.
+
+To vary compression, append to `arch/x86/configs/common.fragment` (uncommitted — the kernel
+then tags `-dirty` and cannot collide with a clean build):
+
+```
+CONFIG_MODULE_COMPRESS_XZ=y       # or leave the flavour's own choice
+CONFIG_MODULE_COMPRESS_ALL=y
+```
+
+A quick synthetic check, useful before spending a full build: `mksquashfs` the module tree both
+ways with EVE's real settings. It predicted core's result to within 30 KB.
+
+```sh
+mksquashfs <tree> out.sqfs -comp xz -Xbcj x86 -b 131072 -no-progress -quiet
+```
+
+## When to revisit
+
+- **The rootfs compressor changes.** The whole result turns on modules and squashfs using the
+  same algorithm. If EVE's squashfs moves to zstd, the conclusions inverts — zstd modules
+  become the cheap option and xz the expensive one.
+- **Container image size starts to matter more than device size** — for example if image pulls
+  dominate a deployment or CI cost, `hwe` with **xz** (not zstd) buys ~491 MiB per image for
+  +1.27 MiB of rootfs.
+- **`core` approaches its cap.** It is already within ~3 MiB of the 295 MiB limit, and
+  compression is already on, so there is no headroom left to buy here.