tools: Add adaptive Bazel wrapper

tools/bazel_adaptive.py runs Bazel with a concrete integer --jobs value,
monitors progress output and memory pressure, stops and resumes bazel
build processes as needed under memory pressure, and if that does not
relieve memory pressure, restarts with lower or higher parallelism as
memory allows. Integer --jobs values are used directly; Bazel-style
HOST_CPUS and HOST_RAM values with optional multipliers, such as
HOST_CPUS*.5 and HOST_RAM*.0002, are resolved once as the maximum
adaptive jobs cap.

The wrapper rewrites bazel output to indicate the number of paused jobs
(if any).  "(13 actions, 12 running)" is shown as "(13 actions, 2 paused,
10 running)" when 2 of the running actions have been paused by the
wrapper.

This wrapper is only modestly faster than builds with carefully tuned
--jobs count and ample available swap space. The main benefit of this
wrapper is that it removes the tuning work and automatically restarts
bazel when if fails for out-of-memory or job termination.

To allow Bazel to get through memory bottlenecks during the build without
unnecessary restarts the build environment should have swap
available. This wrapper was tested on a 24 GB VM with 16 GB swap.

Running the wrapper with BAZEL_ADAPTIVE_DISABLE_PTY=1 defined allows more
of the diagnostic lines to remain visible even when running in an
interactive terminal. Each diagnostic line starts with
"[bazel-adaptive/<XXX>s]", where "<XXX>" is the number of seconds since
the wrapper started.

Written by Codex through an iterative session where codex has run the
included Python tests and has observed the behavior of live runs to
observe the runtime behavior.

Signed-off-by: Jarno Rajahalme <jarno@isovalent.com>

Author: jrajahalme

.gitignore

@@ -51,3 +51,6 @@
 
 # clangd compilation database
 /compile_commands.json
+
+# Python caches
+__pycache__

README.md

@@ -86,6 +86,53 @@ changes refresh it automatically.
 > for the build stages.
 
 
+### Optimizing Memory for Bazel Builds
+
+Sometimes Bazel builds crash due to lack of memory, especially when building
+heavily templated code such as integration tests. The most straightforward
+solution is to lower the number of build jobs via the `--jobs` parameter.
+Makefiles in this repo already scale the number of jobs to the available memory
+when launching Bazel. Depending on the build environment this may leave many or
+even most of the available CPUs underutilized. There are two additional
+strategies to allow more parallel builds: enabling swap and/or using the
+adaptive bazel wrapper.
+
+Enabling swap on the build system allows build jobs to be swapped out when Bazel
+hits a memory bottleneck. There is always some portion of the process memory
+that is not actively used during the build and enabling swap allows that portion
+to be swapped out during the build, freeing memory for the processes that need
+it. But with too many parallel builds the working set of the builders will not
+fit in physical memory, and the build slows to a crawl. Since there are only a
+small number of build actions that take disproportionally memory the build can
+speed up again when the "jumbo" action gets finished. As an example, in a 24GB
+virtual machine with 12 CPUs, `make tests` typically succeeds to build with
+`--jobs=4`, but sometimes even that runs out of memory and `--jobs=3` is
+safer. With swap enabled this build succeeds significantly faster with
+`--jobs=6`.
+
+tools/bazel_adaptive.py is an experimental adaptive wrapper for Bazel. It
+monitors the available memory and swap activity during Bazel builds, and adapts
+with two strategies: On a moment-to-moment basis it can pause and resume Bazel's
+build processes, depending on the available memory and swap activity of the
+build processes. Typically this manages to get past memory bottlenecks,
+especially when enough swap is available so that the stopped processes can be
+swapped out, freeing available memory for the running builders. If Bazel runs
+out of memory and stops, the wrapper will start Bazel again with a lower number
+of jobs so that the build can proceed. This downscaling can go down to 1 job, so
+that the build can finish unattended. Once the memory bottleneck is over the
+wrapper can also upscale bazel by restarting it with a higher number of jobs.
+Example use the adaptive wrapper:
+
+```
+BAZEL=tools/bazel_adaptive.py BAZEL_TEST_OPTS=--test_timeout=100 make tests
+```
+
+This starts bazel with the jobs parameter set to the number of CPUs
+available. Setting `BAZEL_TEST_OPTS` bypasses the job limit built into the
+Makefile, allowing all CPUs to be used for the build. In the example build
+environment this is 12 parallel builders instead of 4.
+
+
 ### Multi-arch builds
 
 Build target architecture can be specified by passing `ARCH`