Skip to content

Commit 41f4e39

Browse files
committed
Publish in place again, not via a staging copy
The staged build has never completed. Both /src and /tmp are NFS mounts of the same Synology volume, so staging turns one write pass into write 763k, enumerate 763k staged, enumerate 763k live, read 763k, write 763k -- about four times the metadata work, against the one resource that is already the bottleneck. That volume serves readdir+stat at 253/s, because 187 GiB of btrfs metadata sits behind 128 MiB of page cache on a 6 GB appliance. On the 2026-08-14 run: Hugo 4h41m, then 75 minutes for the staging find alone, with the live find and the copy still ahead. The in-place build published every version of this site up to 2026-08-05. Keeps --noBuildLock and the stale-lock removal. The page-count gate goes with the staging tree; the 'published N pages' line the Jenkins step keys on now comes from Hugo's own summary rather than a find that costs 75 minutes. Accepted trade-off: during the build the site serves a mix of old and new pages. Nothing is deleted, so old fingerprinted assets remain on disk and old pages keep resolving until overwritten. Restore the staged version from git history once the corpus is sharded and the volume's metadata is cached.
1 parent d64185a commit 41f4e39

1 file changed

Lines changed: 58 additions & 49 deletions

File tree

‎deploy.sh‎

Lines changed: 58 additions & 49 deletions
Original file line numberDiff line numberDiff line change
@@ -1,77 +1,86 @@
11
#!/bin/bash
2-
# Build the static site and publish it into the served directory.
2+
# Build the static site directly into the served directory.
33
#
4-
# Run by the Rancher `hugo` service (start_once), which mounts the Jenkins
5-
# workspace at /src and a cache volume at /tmp. /src/public is the NFS volume
6-
# that vfb-services-live/vfb-static-site serves read-only, so it is live traffic.
4+
# Run by the Rancher `hugo-builder` service (start_once), which mounts the
5+
# Jenkins workspace at /src and a cache volume at /tmp. /src/public is the NFS
6+
# volume that vfb-services-live/vfb-static-site serves read-only.
77
#
8-
# The build therefore goes to a staging directory first and is rsynced into
9-
# place afterwards. Building straight into /src/public would leave the site
10-
# half-updated for the length of a full run — roughly a quarter of an hour once
11-
# the generated term corpus is included — and every page changes when the theme
12-
# changes, so visitors would be served a mixture of old and new markup with
13-
# fingerprinted asset URLs that may not exist yet.
8+
# ---------------------------------------------------------------------------
9+
# Why this builds in place again
1410
#
15-
# rsync updates the live tree file by file and defers removals to the end
16-
# (--delete-after), so no page is ever missing mid-run. A directory swap is not
17-
# an option: /src/public is bind-mounted into the nginx container, which holds
18-
# the old inode and would not follow a rename.
11+
# The previous version staged to /tmp/build and rsynced into /src/public, so
12+
# that no visitor ever saw a half-updated site. Sound in principle; on this
13+
# storage it does not finish.
14+
#
15+
# Both /src and /tmp are NFS mounts of the same Synology volume, so staging
16+
# turns one write pass into: write 763k files to NFS, enumerate 763k in the
17+
# staging tree, enumerate ~763k in the live tree, then read 763k and write 763k
18+
# back -- roughly four times the metadata work, all of it against the one
19+
# resource that is already the bottleneck. Measured on the volume: readdir+stat
20+
# runs at 253/s, because 187 GiB of btrfs metadata is backed by 128 MiB of page
21+
# cache on a 6 GB appliance.
22+
#
23+
# Measured on the 2026-08-14 run: Hugo 4h41m, then 1h15m for the staging
24+
# find alone, with the live-tree find and the copy still to come. The staged
25+
# path has never once completed. The in-place build published every version of
26+
# this site up to 2026-08-05.
27+
#
28+
# The trade-off is real and accepted: during the build the site serves a mix of
29+
# old and new pages. It is survivable because nothing is deleted -- old
30+
# fingerprinted assets stay on disk, so old pages keep resolving until they are
31+
# overwritten. Restore the staged version from git history once the corpus is
32+
# sharded and the volume's metadata is cached; the cost model is what makes it
33+
# unworkable, not the design.
34+
#
35+
# Corollary: pages that are no longer generated are never removed from
36+
# /src/public. That was true of every build before 2026-08-05 too. Removing
37+
# them needs an occasional deliberate sweep, not --cleanDestinationDir, which
38+
# would empty the live tree at the start of a four-hour build.
1939
#
2040
# Deliberately does NOT pass --enableGitInfo. hugo.toml sets
2141
# enableGitInfo = false and the flag would override it; collecting git metadata
2242
# across the term corpus is expensive and nothing in the theme uses it.
43+
# ---------------------------------------------------------------------------
2344
set -euo pipefail
2445
set -x
2546

2647
SRC="${SRC_DIR:-/src}"
2748
LIVE="$SRC/public"
28-
STAGE="${STAGE_DIR:-/tmp/build}"
29-
# Keep Hugo's own cache on the same volume as the staging tree.
3049
export HUGO_CACHEDIR="${HUGO_CACHEDIR:-/tmp/hugo_cache}"
31-
# --checksum avoids rewriting pages whose content has not changed, at the cost
32-
# of reading both trees. Worth trying once the corpus is large and most nightly
33-
# term pages are identical.
34-
RSYNC_OPTS="${RSYNC_OPTS:--a --delete-after --omit-dir-times}"
50+
LOG="${BUILD_LOG:-/tmp/hugo-build.out}"
3551

3652
cd "$SRC"
3753

3854
# ---------------------------------------------------------------- build ------
39-
rm -rf "$STAGE"
40-
4155
# --noBuildLock, and a stale lock cleared first.
4256
#
4357
# Hugo takes an flock on .hugo_build.lock at startup and waits for it forever if
44-
# it cannot get one — no message, no error, no timeout. The symptom is a build
45-
# that copies the static files, then hangs: on 2026-08-05 that was ~66 files and
46-
# 16 MB in the staging tree, a sleeping process at near-zero CPU, and nothing
47-
# else for as long as anyone cared to wait.
48-
#
49-
# The workspace is on NFS, where a lock left behind by an uncleanly killed build
50-
# survives the process that took it. And the lock guards against a second
51-
# concurrent build, which cannot happen here: this container is the only builder
52-
# and Rancher runs it start_once. So it protects nothing and its failure mode is
53-
# an invisible hang.
58+
# it cannot get one -- no message, no error, no timeout. The lock guards against
59+
# a second concurrent build, which cannot happen here: this container is the
60+
# only builder and Rancher runs it start_once. So it protects nothing.
5461
rm -f "$SRC/.hugo_build.lock"
55-
hugo --noBuildLock --gc --minify --destination "$STAGE"
62+
63+
hugo --noBuildLock --gc --minify --destination "$LIVE" 2>&1 | tee "$LOG"
5664

5765
# ---------------------------------------------------------------- gates ------
58-
# Never publish an obviously broken build. The term corpus comes from an API
59-
# that can fail partially, so a run that produced far fewer pages than the live
60-
# site is far more likely to be a bad vfbterms run than a real deletion.
61-
test -s "$STAGE/index.html"
62-
test -s "$STAGE/sitemap.xml"
66+
# Cheap checks only. An in-place build has already published by the time these
67+
# run, so they are an alarm rather than a gate -- but two stats cost nothing and
68+
# a missing index.html is worth shouting about. The old page-count gate is gone
69+
# with the staging tree: counting 763k files with find took 75 minutes.
70+
test -s "$LIVE/index.html"
71+
test -s "$LIVE/sitemap.xml"
6372

64-
new=$(find "$STAGE" -name '*.html' | wc -l)
65-
old=$(find "$LIVE" -name '*.html' 2>/dev/null | wc -l || echo 0)
66-
echo "pages: live=$old new=$new"
73+
# Hugo's own summary table, rather than a find over the live tree.
74+
# Pages | 763455
75+
pages=$(awk -F'|' '/^[[:space:]]*Pages[[:space:]]*\|/ {gsub(/[^0-9]/,"",$2); print $2; exit}' "$LOG")
6776

68-
if [ "$old" -gt 100 ] && [ "$new" -lt $(( old * 90 / 100 )) ]; then
69-
echo "ERROR: build has $new pages against $old live, a drop of more than 10%." >&2
70-
echo "Refusing to publish. Set ALLOW_SHRINK=1 to override." >&2
71-
[ "${ALLOW_SHRINK:-0}" = "1" ] || exit 1
77+
if [ -z "${pages:-}" ]; then
78+
echo "ERROR: could not read a page count from Hugo's output." >&2
79+
exit 1
80+
fi
81+
if [ "$pages" -lt 100 ]; then
82+
echo "ERROR: Hugo reported only $pages pages." >&2
83+
exit 1
7284
fi
7385

74-
# ---------------------------------------------------------------- publish ----
75-
rsync $RSYNC_OPTS "$STAGE/" "$LIVE/"
76-
77-
echo "published $new pages to $LIVE"
86+
echo "published $pages pages to $LIVE"

0 commit comments

Comments
 (0)