Skip to content

Commit 7b10e97

Browse files
grantkeeclaude
andcommitted
fix(update-node): wrapper-aware docker detect/apply + overridable verify window
Docker updates edited the systemd unit, but current installs launch the container from the start wrapper (/opt/telcoin/start-<svc>.sh) since the BLS LoadCredential change -- so --check found no image and prepare/apply failed on every wrapper-based install. detect_current_docker_image and both apply paths now resolve the launch target via tn_node_launch_target and patch whichever file carries the image reference (wrapper, or unit on legacy installs). Also in update-node v1.1.56: - TN_UPDATE_VERIFY_TIMEOUT=<secs> overrides the 45s post-restart health window (numeric-guarded). A fleet-wide simultaneous restart for a protocol-breaking upgrade re-forms quorum slower than a lone restart; the fixed window caused a spurious auto-rollback that would strand the node on the old wire protocol. - backup_unit_file printed its info line into the caller's $(...) capture, so the interactive docker restore path got a garbage backup path. Info now goes to stderr; stdout carries only the path. update-scripts v1.1.62 re-cut with refreshed .sha256 sidecars. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1 parent b88b96f commit 7b10e97

5 files changed

Lines changed: 93 additions & 39 deletions

File tree

README.md

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -876,6 +876,21 @@ prints the exact fix command.
876876
> independently, so entries are titled `<script> vX.Y.Z`. Earlier entries used
877877
> a flat "all scripts bumped to vX.Y.Z" convention.
878878
879+
### update-node v1.1.56
880+
Docker updates now edit the file that actually launches the container. Current docker
881+
installs run `docker run` from the start wrapper (`/opt/telcoin/start-<svc>.sh`) rather
882+
than the unit's ExecStart, but the update path still read and rewrote the systemd unit —
883+
so `--check` reported no current image and prepare/apply failed on every wrapper-based
884+
install. Image detection and both apply paths now resolve the wrapper-vs-legacy-unit
885+
target via `tn_node_launch_target` and patch whichever file carries the image reference.
886+
Two more fixes ride along: `TN_UPDATE_VERIFY_TIMEOUT=<secs>` overrides the 45s
887+
post-restart health window (a fleet-wide simultaneous restart for a wire-protocol-breaking
888+
upgrade re-forms quorum slower than one node restarting, and the default window triggered
889+
a spurious auto-rollback), and the interactive docker apply no longer corrupts its restore
890+
path — `backup_unit_file` printed its info line to stdout inside the caller's `$(...)`
891+
capture, so a rollback would have copied from a garbage path. `update-scripts.sh v1.1.62`
892+
re-cut with refreshed `.sha256` sidecars.
893+
879894
### VPN admin SSH — fix tnadmin key-login lockout + confirm-and-reuse on re-run
880895
`setup-vpn.sh v1.4.0` fixes opted-in nodes going unreachable to maintainers. `tnadmin` was
881896
created with no password and then `passwd -l`'d, leaving the shadow field `!`-locked; Ubuntu

update-node.sh

Lines changed: 75 additions & 36 deletions
Original file line numberDiff line numberDiff line change
@@ -31,11 +31,22 @@ set -uo pipefail
3131
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
3232
source "${SCRIPT_DIR}/lib/common.sh"
3333

34-
readonly SCRIPT_VERSION="1.1.55"
34+
readonly SCRIPT_VERSION="1.1.56"
3535
# GAR_TAGS_URL is provided by lib/common.sh (sourced above). Re-declaring it
3636
# readonly here threw "GAR_TAGS_URL: readonly variable" to stderr, which the UI
3737
# surfaced as "update checks aren't available on this host".
38-
readonly VERIFY_TIMEOUT_SECONDS=45
38+
#
39+
# Post-restart health-verify window (seconds). Overridable via
40+
# TN_UPDATE_VERIFY_TIMEOUT for fleet-orchestrated updates: when every peer
41+
# restarts at once (a wire-protocol-breaking upgrade), quorum takes longer to
42+
# re-form than a single-node restart, and the default window would trigger a
43+
# spurious auto-rollback. Non-numeric values fall back to 45 -- this feeds
44+
# arithmetic under set -u.
45+
if [[ "${TN_UPDATE_VERIFY_TIMEOUT:-}" =~ ^[0-9]+$ ]]; then
46+
readonly VERIFY_TIMEOUT_SECONDS="${TN_UPDATE_VERIFY_TIMEOUT}"
47+
else
48+
readonly VERIFY_TIMEOUT_SECONDS=45
49+
fi
3950

4051
NODE_TYPE=""
4152
SERVICE_NAME=""
@@ -125,10 +136,24 @@ detect_network() {
125136
echo ""
126137
}
127138

139+
# Resolve the file that carries the docker image reference on this install:
140+
# the start wrapper (${DEFAULT_INSTALL_DIR}/start-<svc>.sh) on current installs
141+
# (the BLS LoadCredential change moved `docker run` out of the unit's ExecStart),
142+
# or the systemd unit on legacy installs that still inline it. Delegates to
143+
# tn_node_launch_target (lib/common.sh). Echoes the path; returns 1 if no node
144+
# service is present or the file is missing.
145+
docker_launch_file() {
146+
local line file
147+
line="$(tn_node_launch_target)" || return 1
148+
file="${line##* }" # last field; install paths never contain spaces
149+
[[ -f "$file" ]] || return 1
150+
echo "$file"
151+
}
152+
128153
detect_current_docker_image() {
129-
local unit="/etc/systemd/system/${SERVICE_NAME}.service"
130-
[[ -f "$unit" ]] || return 1
131-
grep -oE 'us-docker[^ ]+|gcr\.io[^ ]+|ghcr\.io[^ ]+' "$unit" | head -1
154+
local file
155+
file="$(docker_launch_file)" || return 1
156+
grep -oE 'us-docker[^ ]+|gcr\.io[^ ]+|ghcr\.io[^ ]+' "$file" | head -1
132157
}
133158

134159
# git describe-style summary of the current source ref + commit
@@ -199,14 +224,16 @@ start_service() {
199224
systemctl start "$SERVICE_NAME"
200225
}
201226

202-
# Backup the systemd unit file with a timestamped sibling. Echoes backup path.
227+
# Backup a launch config file (unit or wrapper) with a timestamped sibling.
228+
# Echoes ONLY the backup path on stdout -- callers capture it via $(...), so the
229+
# human-readable info line must go to stderr or it corrupts the captured path.
203230
backup_unit_file() {
204231
local file="$1"
205232
local ts backup
206233
ts=$(date -u '+%Y%m%d-%H%M%S')
207234
backup="${file}.bak.${ts}"
208235
cp -p "$file" "$backup" || return 1
209-
print_info "Unit file backup: ${backup}"
236+
print_info "Launch config backup: ${backup}" >&2
210237
echo "$backup"
211238
}
212239

@@ -278,7 +305,7 @@ prepare_docker_update() {
278305

279306
local current_image
280307
current_image=$(detect_current_docker_image) || {
281-
print_error "Could not read current Docker image from ${SERVICE_NAME}.service"
308+
print_error "Could not read current Docker image from the ${SERVICE_NAME} launch config (wrapper or unit)"
282309
return 1
283310
}
284311
print_info "Current image: ${current_image}"
@@ -323,47 +350,54 @@ apply_docker_update() {
323350
echo ""
324351
validator_downtime_warning_if_applicable || return 1
325352

326-
local unit="/etc/systemd/system/${SERVICE_NAME}.service"
353+
# The image reference lives in the start wrapper on current installs and in
354+
# the systemd unit only on legacy ones -- resolve whichever this node uses.
355+
local launch_file
356+
launch_file=$(docker_launch_file) || {
357+
print_error "Could not resolve the docker launch config (wrapper or unit) for ${SERVICE_NAME}."
358+
return 1
359+
}
327360
local backup
328-
backup=$(backup_unit_file "$unit") || return 1
361+
backup=$(backup_unit_file "$launch_file") || return 1
329362

330-
# Hash the unit file before/after the substitution so we can detect
363+
# Hash the launch config before/after the substitution so we can detect
331364
# "perl did nothing" -- symmetric with the source-binary hash check
332365
# added in v1.1.41. Avoids reporting a successful update when the
333-
# unit file was actually unchanged.
366+
# file was actually unchanged.
334367
local pre_unit_hash post_unit_hash
335-
pre_unit_hash=$(sha256sum "$unit" | awk '{print $1}')
368+
pre_unit_hash=$(sha256sum "$launch_file" | awk '{print $1}')
336369

337370
print_step "Stopping ${SERVICE_NAME}..."
338371
wait_for_service_stopped "$SERVICE_NAME"
339372

340-
print_step "Updating unit file image reference..."
341-
perl -i -pe "s|\Q${old_image}\E|${new_image}|g" "$unit"
373+
print_step "Updating image reference in ${launch_file}..."
374+
perl -i -pe "s|\Q${old_image}\E|${new_image}|g" "$launch_file"
375+
# daemon-reload matters on legacy unit installs; harmless for the wrapper.
342376
systemctl daemon-reload
343377

344-
post_unit_hash=$(sha256sum "$unit" | awk '{print $1}')
378+
post_unit_hash=$(sha256sum "$launch_file" | awk '{print $1}')
345379
if [[ "$pre_unit_hash" == "$post_unit_hash" ]]; then
346-
print_error "Unit file is unchanged after edit -- old image string not found."
380+
print_error "Launch config is unchanged after edit -- old image string not found."
347381
print_info " Expected to replace: ${old_image}"
348-
print_info " Unit file: ${unit}"
382+
print_info " Launch config: ${launch_file}"
349383
print_info "Restoring from backup and aborting."
350-
cp -p "$backup" "$unit"
384+
cp -p "$backup" "$launch_file"
351385
systemctl daemon-reload
352386
start_service 2>/dev/null || true
353387
return 1
354388
fi
355389
# Confirm the new image actually appears in the file (defends against a
356390
# perl substitution that replaced the wrong text).
357-
if ! grep -qF "$new_image" "$unit"; then
358-
print_error "New image string not present in unit file after edit."
391+
if ! grep -qF "$new_image" "$launch_file"; then
392+
print_error "New image string not present in launch config after edit."
359393
print_info " Expected to find: ${new_image}"
360394
print_info "Restoring from backup and aborting."
361-
cp -p "$backup" "$unit"
395+
cp -p "$backup" "$launch_file"
362396
systemctl daemon-reload
363397
start_service 2>/dev/null || true
364398
return 1
365399
fi
366-
print_ok "Unit file updated: ${pre_unit_hash:0:12}... -> ${post_unit_hash:0:12}..."
400+
print_ok "Launch config updated: ${pre_unit_hash:0:12}... -> ${post_unit_hash:0:12}..."
367401

368402
print_step "Starting ${SERVICE_NAME} on new image..."
369403
start_service
@@ -385,7 +419,7 @@ apply_docker_update() {
385419
if confirm "Roll back to previous image (${old_image})?"; then
386420
print_step "Rolling back..."
387421
wait_for_service_stopped "$SERVICE_NAME"
388-
cp -p "$backup" "$unit"
422+
cp -p "$backup" "$launch_file"
389423
systemctl daemon-reload
390424
start_service
391425
if verify_health_after_restart; then
@@ -761,7 +795,7 @@ validator_downtime_warning_if_applicable() {
761795
pick_docker_version() {
762796
local current_image
763797
current_image=$(detect_current_docker_image) || {
764-
print_error "Could not read current Docker image from ${SERVICE_NAME}.service" >&2
798+
print_error "Could not read current Docker image from the ${SERVICE_NAME} launch config (wrapper or unit)" >&2
765799
return 1
766800
}
767801
local current_tag="${current_image##*:}"
@@ -1170,24 +1204,29 @@ json_apply_docker() {
11701204
if [[ -z "$old_image" || -z "$new_image" ]]; then
11711205
json_event error "pending state is incomplete -- cannot apply"; return 1
11721206
fi
1173-
local unit="/etc/systemd/system/${SERVICE_NAME}.service"
1207+
# The image reference lives in the start wrapper on current installs and in
1208+
# the systemd unit only on legacy ones -- resolve whichever this node uses.
1209+
local launch_file
1210+
launch_file=$(docker_launch_file) || {
1211+
json_event error "could not resolve docker launch config (wrapper or unit)"; return 1; }
11741212
local ts backup
11751213
ts=$(date -u '+%Y%m%d-%H%M%S')
1176-
backup="${unit}.bak.${ts}"
1177-
cp -p "$unit" "$backup" || { json_event error "could not back up unit file"; return 1; }
1214+
backup="${launch_file}.bak.${ts}"
1215+
cp -p "$launch_file" "$backup" || { json_event error "could not back up launch config"; return 1; }
11781216
local pre_hash post_hash
1179-
pre_hash=$(sha256sum "$unit" | awk '{print $1}')
1217+
pre_hash=$(sha256sum "$launch_file" | awk '{print $1}')
11801218

11811219
json_event step "Stopping ${SERVICE_NAME}"
11821220
wait_for_service_stopped "$SERVICE_NAME"
11831221

1184-
json_event step "Updating unit file image reference"
1185-
perl -i -pe "s|\Q${old_image}\E|${new_image}|g" "$unit"
1222+
json_event step "Updating image reference in ${launch_file}"
1223+
perl -i -pe "s|\Q${old_image}\E|${new_image}|g" "$launch_file"
1224+
# daemon-reload matters on legacy unit installs; harmless for the wrapper.
11861225
systemctl daemon-reload
1187-
post_hash=$(sha256sum "$unit" | awk '{print $1}')
1188-
if [[ "$pre_hash" == "$post_hash" ]] || ! grep -qF "$new_image" "$unit"; then
1189-
json_event error "unit file image not updated -- restoring backup"
1190-
cp -p "$backup" "$unit"; systemctl daemon-reload; start_service 2>/dev/null || true
1226+
post_hash=$(sha256sum "$launch_file" | awk '{print $1}')
1227+
if [[ "$pre_hash" == "$post_hash" ]] || ! grep -qF "$new_image" "$launch_file"; then
1228+
json_event error "launch config image not updated -- restoring backup"
1229+
cp -p "$backup" "$launch_file"; systemctl daemon-reload; start_service 2>/dev/null || true
11911230
return 1
11921231
fi
11931232

@@ -1203,7 +1242,7 @@ json_apply_docker() {
12031242

12041243
json_event step "Health check failed -- rolling back to previous image"
12051244
wait_for_service_stopped "$SERVICE_NAME"
1206-
cp -p "$backup" "$unit"; systemctl daemon-reload; start_service
1245+
cp -p "$backup" "$launch_file"; systemctl daemon-reload; start_service
12071246
if verify_health_after_restart; then
12081247
clear_pending_state
12091248
json_emit "{\"event\":\"done\",\"ok\":false,\"phase\":\"apply\",\"rolled_back\":true,\"msg\":\"health check failed; rolled back to previous image\"}"

update-node.sh.sha256

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1 @@
1-
baf49d8b6c805817b37a11a15be1d2d2458acc22e9d128c7ee92c939dfc462fc update-node.sh
1+
5a63af0a87cd08de36f6cc227ebde712e0c87bb871eba6af0fdbe5af1ec1d8c2 update-node.sh

update-scripts.sh

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,7 @@ SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
1919
# shellcheck source=lib/fallback.sh
2020
source "${SCRIPT_DIR}/lib/fallback.sh" 2>/dev/null || true
2121

22-
readonly SCRIPT_VERSION="1.1.61"
22+
readonly SCRIPT_VERSION="1.1.62"
2323
readonly GITHUB_RAW="https://raw.githubusercontent.com/Telcoin-Association/tn-node-deployment/main"
2424

2525
# Colours

update-scripts.sh.sha256

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1 @@
1-
210fedc84d8a851e6e38152ecbbe50cec2a261f0db274d478122b0049901f0e7 update-scripts.sh
1+
5bd2dda788cefd1f1cec3315a5c2b03f67df56950c020c3a4149438b48ec70d1 update-scripts.sh

0 commit comments

Comments
 (0)