feat(ops): compose-managed vm-watchdog sidecar, replacing the host cron watchdog

psyb0t's review: the crash-recovery failure mode is real, but a host cron job
is the wrong vehicle — it needs a machine-specific checkout path and cron setup,
and the fixed cooldown cannot prevent a restart loop. This replaces
scripts/watchdog.sh with a Compose-managed vm-watchdog sidecar that ships in
docker compose up -d.

- scripts/vm-watchdog.py: stdlib-only loop that polls Docker health through the
  mounted unix socket. Scoped to this compose project (self-discovered from its
  own labels) plus the dockurr/windows image, so it only ever restarts VM
  containers. Uses Docker's State.Health.FailingStreak as the source of truth,
  keeps a per-container state record on a named volume (last restart, attempts,
  healthy-since), enforces exponential backoff (5m -> 15m -> 1h), stops after
  WATCHDOG_MAX_ATTEMPTS failed recoveries with a loud log, and resets the
  attempt budget only after the VM has stayed healthy for a meaningful period.
  Fixes the two correctness issues in the old script by construction: the loop
  never exits with a "recovered" error, and backoff is exponential, not a fixed
  cooldown. Uses docker restart (not recreate), preserving the owner container
  ID and therefore wickworks sidecar netns attachment.
- docker-compose.yml.j2/.example: vm-watchdog service (socket + script mounts,
  named vm-watchdog-state volume), like the log-rotator, one per project.
- tests/test_vm_watchdog.py: behavioural coverage for healthy/starting
  exclusion, sustained-unhealthy restart, image/label scoping, exponential
  backoff, bounded retries, reset after stable health, dry-run, and project
  self-discovery — against a fake Docker transport, so it runs offline.
- docs/operations.md: Auto-recovery section rewritten for the sidecar; cron
  instructions removed.
- Dockerfile.test: COPY the new script so the offline suite exercises it.

Author: Marinski

Dockerfile.test

@@ -25,7 +25,7 @@ COPY tests ./tests
 # docker-compose.yml.j2 is here because the compose-generation test renders the
 # REAL template — a stub would assert nothing about what actually ships.
 COPY requirements-api.txt requirements-mcpunifier.txt docker-compose.yml.example docker-compose.yml.j2 run.sh ./
-COPY scripts/config_helper.py scripts/start.bat scripts/check_health.py scripts/healthcheck.sh scripts/verify_binaries.py scripts/wickworks-healthcheck.py ./scripts/
+COPY scripts/config_helper.py scripts/start.bat scripts/check_health.py scripts/healthcheck.sh scripts/verify_binaries.py scripts/wickworks-healthcheck.py scripts/vm-watchdog.py ./scripts/
 COPY assets/binaries.lock.json ./assets/
 
 ENV PYTHONPATH=/app

docker-compose.yml.example

@@ -91,6 +91,30 @@ services:
       - ./scripts/rotate-logs.sh:/rotate.sh:ro
     command: ["sh", "/rotate.sh"]
 
+  # VM crash watchdog. dockurr/windows keeps the container up while the
+  # Windows guest may have crashed internally, so restart: unless-stopped
+  # never fires and every terminal API in that VM stays dead. This sidecar
+  # polls Docker health through the socket and restarts a VM only after its
+  # health has stayed unhealthy for a sustained FailingStreak, with
+  # exponential backoff and bounded retries; state lives on a named volume.
+  # Runs inside the compose project (docker compose up -d), no host cron.
+  # See scripts/vm-watchdog.py.
+  vm-watchdog:
+    image: python:3.12-alpine
+    restart: unless-stopped
+    command: ["python", "-u", "/vm-watchdog.py"]
+    volumes:
+      - /var/run/docker.sock:/var/run/docker.sock
+      - ./scripts/vm-watchdog.py:/vm-watchdog.py:ro
+      - vm-watchdog-state:/state
+    environment:
+      WATCHDOG_STATE_DIR: /state
+    logging:
+      driver: json-file
+      options:
+        max-size: "10m"
+        max-file: "3"
+
   # nginx is the single entry point for all terminal APIs. Routes
   # /<broker>/<account>/... to mt5:<terminal_port> (per-terminal Python
   # API process inside the Windows VM, reachable via mt5 container's
@@ -198,3 +222,8 @@ services:
   #     - NET_RAW
   #   depends_on:
   #     - nginx
+
+volumes:
+  # Persistent per-container watchdog state (last restart, attempts,
+  # healthy-since) so backoff survives the watchdog's own restarts.
+  vm-watchdog-state:

docker-compose.yml.j2

@@ -120,6 +120,30 @@ services:
       - ./scripts/rotate-logs.sh:/rotate.sh:ro
     command: ["sh", "/rotate.sh"]
 
+  # VM crash watchdog. dockurr/windows keeps the container up while the
+  # Windows guest may have crashed internally, so restart: unless-stopped
+  # never fires and every terminal API in that VM stays dead. This sidecar
+  # polls Docker health through the socket and restarts a VM only after its
+  # health has stayed unhealthy for a sustained FailingStreak, with
+  # exponential backoff and bounded retries; state lives on a named volume.
+  # Runs inside the compose project (docker compose up -d), no host cron.
+  # See scripts/vm-watchdog.py.
+  vm-watchdog:
+    image: python:3.12-alpine
+    restart: unless-stopped
+    command: ["python", "-u", "/vm-watchdog.py"]
+    volumes:
+      - /var/run/docker.sock:/var/run/docker.sock
+      - ./scripts/vm-watchdog.py:/vm-watchdog.py:ro
+      - vm-watchdog-state:/state
+    environment:
+      WATCHDOG_STATE_DIR: /state
+    logging:
+      driver: json-file
+      options:
+        max-size: "10m"
+        max-file: "3"
+
 {% if enable_mcpunifier|default(true) %}
   # Unified MCP endpoint. One MCP session that reaches every terminal, with
   # broker/account as tool parameters, instead of one endpoint per terminal.
@@ -229,3 +253,8 @@ services:
   #     - NET_RAW
   #   depends_on:
   #     - nginx
+
+volumes:
+  # Persistent per-container watchdog state (last restart, attempts,
+  # healthy-since) so backoff survives the watchdog's own restarts.
+  vm-watchdog-state:

docs/operations.md

@@ -262,35 +262,45 @@ OOM — leaves the **container** up while the **API** is dead, so
 
 ### VM crash watchdog
 
-`scripts/watchdog.sh` is the host-side recovery for exactly that case. Run it
-from cron or systemd every few minutes; it is idempotent and safe to overlap:
-
-```bash
-# every 5 minutes
-*/5 * * * * /home/monster/apps/mt-backtest-manager/docker/mt5-httpapi/scripts/watchdog.sh >> /var/log/mt5-httpapi-watchdog.log 2>&1
-```
+`vm-watchdog` is a Compose-managed sidecar for exactly that case. It is part of
+the project (`docker compose up -d` brings it up with everything else) — no
+host cron, no systemd unit, no machine-specific checkout path. It polls Docker
+health through the mounted unix socket and uses Docker's own
+`State.Health.FailingStreak` as the source of truth.
 
 Behavior:
 
-- Discovers VM containers by compose labels
-  (`com.docker.compose.project=mt5-httpapi` + `com.docker.compose.service`) and
-  verifies the image (`dockurr/windows`), so it only ever restarts the VM
-  containers — never nginx, wickworks, the log rotator, or other sidecars.
+- Scopes itself to this Compose project (`com.docker.compose.project`,
+  discovered from its own container labels) and to the `dockurr/windows` VM
+  image, so it only ever restarts the VM containers — never nginx, wickworks,
+  the log rotator, or other sidecars.
 - Restarts a container **only** after its Docker health has stayed `unhealthy`
   for `WATCHDOG_MIN_FAILING_STREAK` consecutive healthcheck failures (default
   `10`, i.e. ~5 minutes at the default 30s interval). A container that is
   healthy or still starting is never touched, so running backtests on a working
   VM are never interrupted — the healthcheck stays green the whole time a
   terminal is serving.
-- Refuses to restart a container that was (re)started less than
-  `WATCHDOG_RESTART_COOLDOWN_SECONDS` ago (default `300`), so a VM that crashes
-  again immediately after recovery is not restarted into a loop.
+- Keeps a tiny state record per container on a named volume
+  (`/state/<container-id>.json`): last restart, attempt count, and when the VM
+  was last observed healthy.
+- Enforces exponential backoff between recovery attempts
+  (`WATCHDOG_BACKOFF_ATTEMPTS`, default `300,900,3600` — 5m → 15m → 1h), so a
+  VM that crashes again immediately after recovery is not restarted into a
+  loop.
+- Stops after `WATCHDOG_MAX_ATTEMPTS` consecutive failed recoveries (default
+  `3`) and logs loudly, instead of threshing forever.
+- Resets the attempt budget only after the VM has stayed healthy for
+  `WATCHDOG_RESET_SECONDS` (default `1800`), so a VM that recovered then crashed
+  later gets a fresh budget.
 - `WATCHDOG_DRY_RUN=1` (or `--dry-run`) prints what it would do without
-  touching any container — useful to sanity-check a cron line before enabling it.
-
-Environment overrides: `WATCHDOG_COMPOSE_PROJECT`, `WATCHDOG_IMAGE_FILTER`,
-`WATCHDOG_MIN_FAILING_STREAK`, `WATCHDOG_RESTART_COOLDOWN_SECONDS`,
-`WATCHDOG_DRY_RUN`.
+  touching any container.
+
+Environment overrides: `WATCHDOG_INTERVAL_SECONDS`, `WATCHDOG_MIN_FAILING_STREAK`,
+`WATCHDOG_IMAGE_FILTER`, `WATCHDOG_BACKOFF_ATTEMPTS`, `WATCHDOG_MAX_ATTEMPTS`,
+`WATCHDOG_RESET_SECONDS`, `WATCHDOG_COMPOSE_PROJECT`, `WATCHDOG_STATE_DIR`,
+`WATCHDOG_DOCKER_SOCKET`, `WATCHDOG_DRY_RUN`. It uses `docker restart` (not
+recreate), so the VM's container ID — and therefore a wickworks sidecar's netns
+attachment — is preserved.
 
 This complements the in-VM `MT5AutoReboot` scheduled task, which reboots on a
 fixed timer and can interrupt long-running backtests; operators who disable that