Skip to content

Commit 77102ad

Browse files
MatusBekeclaude
MatusBeke
and
claude
committed
Add frontend healthcheck and an autoheal sidecar with a restart budget
Docker does not restart a container just because its healthcheck says unhealthy - the restart policy only reacts to the main process exiting. So a container can sit unhealthy indefinitely while `docker ps` shows it running. The optional autoheal overlay closes that gap for containers labelled autoheal=true. It is deliberately NOT willfarrell/autoheal. That image has no restart budget: if a container can never become healthy it restarts it forever. That happened three times while building this - once on the backend (a config/version mismatch meant Spring never booted) and twice on the frontend (`ng serve` needs around 15 minutes to build and the start_period was too short, so the build was killed mid-flight). In every case the restarts made the problem harder to diagnose, because the container kept dying under the person reading its logs. This sidecar restarts at most AUTOHEAL_MAX_RESTARTS times per AUTOHEAL_WINDOW_SECONDS, then gives up, logs once, and leaves the container unhealthy for a human. The budget resets when the container reports healthy. It stays in a separate overlay rather than the base compose because it mounts docker.sock, which grants effective root on the host - that should be an explicit choice, not something a plain `docker compose up` turns on. The frontend probe targets / and not: /app/health - a pure proxy to the backend actuator (server.ts healthCheck), so it reports the backend's health, not the UI's. It would mark the UI unhealthy whenever the backend is down, and restarting the UI cannot fix the backend. /robots.txt - that express route only exists in the SSR server, i.e. the dist image. This overlay uses the development image, where `ng serve` returns 404 for it. Measured: / -> 200 in 35ms, /robots.txt -> 404. start_period is 900s because `ng serve` compiles bundles at container start and does not answer the first request for many minutes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
1 parent 28da822 commit 77102ad

2 files changed

Lines changed: 141 additions & 1 deletion

File tree

‎dspace/src/main/docker-compose/docker-compose-angular.yml‎

Lines changed: 30 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,8 @@ services:
1616
dspace-angular:
1717
container_name: dspace-angular
1818
depends_on:
19-
- dspace
19+
dspace:
20+
condition: service_healthy
2021
environment:
2122
DSPACE_UI_SSL: 'false'
2223
DSPACE_UI_HOST: dspace-angular
@@ -34,3 +35,31 @@ services:
3435
target: 9876
3536
stdin_open: true
3637
tty: true
38+
# Probe choice, and why it is not the obvious one:
39+
#
40+
# NOT /app/health - that endpoint is a pure proxy to the backend actuator (see
41+
# server.ts healthCheck()), so it reports the BACKEND's health, not this
42+
# container's. It would mark the UI unhealthy whenever the backend is down, and
43+
# restarting the UI cannot fix the backend, so autoheal would loop. It also
44+
# forwards the backend's HTTP 200-on-DOWN.
45+
#
46+
# NOT /robots.txt either - that express route lives in the SSR server (server.ts)
47+
# which only runs in the *dist* image. This overlay uses dspace/dspace-angular,
48+
# the DEVELOPMENT image: it runs `ng serve`, which returns 404 for /robots.txt.
49+
# Measured on this image: / -> 200 in 35ms, /robots.txt -> 404.
50+
# If you switch this overlay to dspace-angular-dist, /robots.txt becomes the
51+
# better probe - it is served from a template with no backend call.
52+
#
53+
# start_period is large on purpose: `ng serve` compiles the bundles at container
54+
# start and does not answer the first request for many minutes. A short value here
55+
# means autoheal kills the build and the container never finishes booting - we hit
56+
# exactly that during testing.
57+
healthcheck:
58+
test: ["CMD-SHELL", "node -e \"const r=require('http').get({host:'127.0.0.1',port:4000,path:'/'},s=>process.exit(s.statusCode===200?0:1));r.setTimeout(8000,()=>{r.destroy();process.exit(1)});r.on('error',()=>process.exit(1))\""]
59+
interval: 30s
60+
timeout: 10s
61+
retries: 3
62+
start_period: 900s
63+
labels:
64+
# Consumed by the optional autoheal sidecar, see docker-compose-autoheal.yml
65+
autoheal: "true"

‎dspace/src/main/docker-compose/docker-compose-autoheal.yml‎

Lines changed: 111 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,111 @@
1+
#
2+
# The contents of this file are subject to the license and copyright
3+
# detailed in the LICENSE and NOTICE files at the root of the source
4+
# tree and available online at
5+
#
6+
# http://www.dspace.org/license/
7+
#
8+
9+
# Optional sidecar that restarts unhealthy containers - with a restart budget.
10+
#
11+
# WHY THIS EXISTS
12+
# Docker does NOT restart a container just because its healthcheck says "unhealthy".
13+
# The `restart` policy only reacts to the main process exiting. A container can sit
14+
# unhealthy indefinitely while `docker ps` still shows it running - exactly the
15+
# "nothing crashed but something inside was broken" failure mode.
16+
#
17+
# WHY NOT willfarrell/autoheal
18+
# That image has no restart budget: if a container can never become healthy, it
19+
# restarts it forever. We hit this three times while building this setup - once on
20+
# the backend (a config/version mismatch meant Spring never booted) and twice on the
21+
# frontend (`ng serve` needs ~15 min to build; the healthcheck's start_period was too
22+
# short, so the build was killed mid-flight and could never finish). In both cases the
23+
# restarts made the problem HARDER to diagnose, because the container kept dying
24+
# under the person reading its logs.
25+
#
26+
# WHAT THIS DOES INSTEAD
27+
# Restarts an unhealthy container at most MAX_RESTARTS times within WINDOW_SECONDS.
28+
# Past that it gives up, logs a loud one-off message, and leaves the container
29+
# unhealthy so a human can look at it. The budget resets once the container reports
30+
# healthy again.
31+
#
32+
# SECURITY: mounting /var/run/docker.sock grants this container effective root on the
33+
# host. Acceptable for local development; for production either accept it deliberately,
34+
# put a restricted socket proxy in front of it, or move to an orchestrator with
35+
# built-in restart semantics (Swarm, Kubernetes livenessProbe + backoff).
36+
#
37+
# Usage:
38+
# docker compose -p d7 -f docker-compose.yml \
39+
# -f dspace/src/main/docker-compose/docker-compose-autoheal.yml up -d
40+
41+
networks:
42+
# Default to using network named 'dspacenet' from docker-compose.yml.
43+
default:
44+
name: ${COMPOSE_PROJECT_NAME}_dspacenet
45+
external: true
46+
services:
47+
autoheal:
48+
container_name: autoheal
49+
image: docker:27-cli
50+
restart: always
51+
environment:
52+
# Only containers carrying this label are eligible
53+
AUTOHEAL_LABEL: ${AUTOHEAL_LABEL:-autoheal}
54+
# Seconds between polls of the Docker API
55+
AUTOHEAL_INTERVAL: ${AUTOHEAL_INTERVAL:-10}
56+
# Restart budget: at most MAX_RESTARTS restarts per WINDOW_SECONDS, per container
57+
AUTOHEAL_MAX_RESTARTS: ${AUTOHEAL_MAX_RESTARTS:-3}
58+
AUTOHEAL_WINDOW_SECONDS: ${AUTOHEAL_WINDOW_SECONDS:-3600}
59+
volumes:
60+
- /var/run/docker.sock:/var/run/docker.sock
61+
entrypoint:
62+
- /bin/sh
63+
- '-c'
64+
- |
65+
set -eu
66+
STATE=/tmp/autoheal
67+
mkdir -p "$$STATE"
68+
log() { echo "$$(date '+%Y-%m-%d %H:%M:%S') $$*"; }
69+
log "watching label $$AUTOHEAL_LABEL=true; budget $$AUTOHEAL_MAX_RESTARTS restarts / $${AUTOHEAL_WINDOW_SECONDS}s"
70+
while true; do
71+
now=$$(date +%s)
72+
73+
# Reset the budget for anything that recovered.
74+
for cid in $$(docker ps -q --filter "label=$$AUTOHEAL_LABEL=true" --filter "health=healthy"); do
75+
name=$$(docker inspect -f '{{.Name}}' "$$cid" | tr -d '/')
76+
if [ -f "$$STATE/$$name.gaveup" ]; then log "$$name recovered - restart budget reset"; fi
77+
rm -f "$$STATE/$$name" "$$STATE/$$name.gaveup"
78+
done
79+
80+
for cid in $$(docker ps -q --filter "label=$$AUTOHEAL_LABEL=true" --filter "health=unhealthy"); do
81+
name=$$(docker inspect -f '{{.Name}}' "$$cid" | tr -d '/')
82+
f="$$STATE/$$name"
83+
84+
# Drop restart timestamps that fell out of the window.
85+
if [ -f "$$f" ]; then
86+
awk -v now="$$now" -v w="$$AUTOHEAL_WINDOW_SECONDS" 'now-$$1 < w' "$$f" > "$$f.tmp" || true
87+
mv "$$f.tmp" "$$f"
88+
else
89+
: > "$$f"
90+
fi
91+
count=$$(wc -l < "$$f" | tr -d ' ')
92+
93+
if [ "$$count" -ge "$$AUTOHEAL_MAX_RESTARTS" ]; then
94+
# Loud once, then quiet - so the log stays readable while someone debugs.
95+
if [ ! -f "$$f.gaveup" ]; then
96+
touch "$$f.gaveup"
97+
log "GIVING UP on $$name: $$count restarts in the last $${AUTOHEAL_WINDOW_SECONDS}s did not help."
98+
log "GIVING UP on $$name: leaving it unhealthy for investigation. Restarting is not fixing this - check its logs."
99+
fi
100+
continue
101+
fi
102+
103+
log "$$name is unhealthy - restarting ($$((count+1))/$$AUTOHEAL_MAX_RESTARTS in window)"
104+
if docker restart -t 10 "$$cid" >/dev/null 2>&1; then
105+
echo "$$now" >> "$$f"
106+
else
107+
log "$$name restart FAILED"
108+
fi
109+
done
110+
sleep "$$AUTOHEAL_INTERVAL"
111+
done

0 commit comments

Comments
 (0)