Skip to content

Commit 2388142

Browse files
KasinhouMatus Kasakclaude
authored
JCU/fix(deploy): run DB migration in one-shot container (fix exit 137) (#1345)
* JCU/fix(deploy): run DB migration in one-shot container (fix exit 137) Replaces the debug instrumentation added in #1344 with the actual fix. Problem ------- On import (ERASE_DB + IMPORT of a DSpace-6 dump) the "run database migration" step failed with exit code 137 (SIGKILL). It was NOT a memory/OOM issue (MemLimit=0, no dmesg OOM, no OutOfMemoryError, JVM peaked ~1GB with 7GB+ free) and NOT a broken migration (Flyway applied migrations cleanly up to 9.0.x when run in isolation). Root cause ---------- The dspace container entrypoint auto-runs `dspace database migrate` (WITHOUT `ignored`) and then starts the REST webapp. On a freshly imported DSpace-6 dump the "Ignored" XMLWorkflow migrations (5.0.2014.11.04, 6.0.2015.09.01) are not applied, so the webapp boots against an un-migrated DB, crashes, and the restart policy restarts the container. That restart SIGKILLs the `docker exec ... migrate ignored` running inside it -> exit 137. (It only "worked" manually because the DB was already near-current, so migrate finished in <1s before any restart.) Fix --- Run the migration in a separate one-shot container instead of exec-ing into the live app container: 1. docker stop the app -> nothing can crash/restart under us 2. docker run --rm a throwaway container (PID 1 = migrate, no webapp, no restart policy) reusing the app container's image / network / volumes / env (via docker inspect) so it hits the same DB 3. docker start the app -> its entrypoint migrate is now a no-op Verified on dev-6/8593: one-shot migration completed, `dspace database info` shows schema fully up to date (all Success through 9.0.2025.03.12). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * JCU/refactor(deploy): run one-off migration via docker compose, not docker run Rewrites the migration step to use `docker compose run` (consistent with the deploy step's tooling) instead of `docker run` + `docker inspect`. - add `actions/checkout` to import-3 so the docker/ compose files are available (the job previously had no checkout). - run the migration as a one-off compose container of the `dspace` service with an overridden entrypoint (migrate only, no webapp), using the same compose files/env-file as the deploy step. Same behaviour as before (app stopped first, no restart loop to SIGKILL the migration), just compose-native. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Matus Kasak <matus.kasak@dataquest.sk> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent 00497a0 commit 2388142

1 file changed

Lines changed: 23 additions & 50 deletions

File tree

‎.github/workflows/deploy.yml‎

Lines changed: 23 additions & 50 deletions
Original file line numberDiff line numberDiff line change
@@ -96,6 +96,8 @@ jobs:
9696
FNAME: old_dump.sql
9797
ADMIN_PASSWORD: ${{ secrets.DSPACE_ADMIN_PASSWORD }}
9898
steps:
99+
- uses: actions/checkout@v6
100+
99101
- name: vanilla import
100102
run: |
101103
export DATADIR=$CONFIG_PATH/${{inputs.INSTANCE}}/dump/$FNAME
@@ -122,62 +124,33 @@ jobs:
122124
- name: run database migration
123125
run: |
124126
export DNAME=dspace${{inputs.INSTANCE}}
125-
export DDBNAME=dspacedb${{inputs.INSTANCE}}
126-
export SYNCSEQSETUP=$CONFIG_PATH/${{inputs.INSTANCE}}/$SYNCSEQFNAME
127-
128-
echo "Starting DSpace container for migration..."
129-
docker start $DNAME
130-
131-
echo "Waiting longer for DSpace to be fully ready and locks to clear..."
132-
sleep 20
133-
134-
# ---------------------------------------------------------------
135-
# DEBUG: baseline resource state BEFORE migration
136-
# ---------------------------------------------------------------
137-
echo "===== DEBUG: host memory (free -h) BEFORE ====="
138-
free -h || true
139-
echo "===== DEBUG: container memory limit / swap ====="
140-
docker inspect $DNAME --format 'MemLimit(bytes)={{.HostConfig.Memory}} MemSwap={{.HostConfig.MemorySwap}} OomKillDisable={{.HostConfig.OomKillDisable}}' || true
141-
echo "===== DEBUG: docker stats snapshot BEFORE ====="
142-
docker stats --no-stream || true
143-
echo "===== DEBUG: java processes already running in container ====="
144-
docker exec $DNAME /bin/bash -c "ps -eo pid,rss,cmd --sort=-rss | head -20" || true
127+
export ENVFILE=$CONFIG_PATH/${{inputs.INSTANCE}}/.env
128+
export OVERLAY=$CONFIG_PATH/${{inputs.INSTANCE}}
145129
146130
# ---------------------------------------------------------------
147-
# DEBUG: sample docker stats every 2s in the background during migrate
131+
# Why a one-off `docker compose run` instead of `docker exec $DNAME migrate`:
132+
# The dspace entrypoint auto-runs `dspace database migrate` (WITHOUT `ignored`)
133+
# and then starts the REST webapp. On a freshly imported DSpace-6 dump the
134+
# "Ignored" XMLWorkflow migrations are NOT applied, so the webapp boots against
135+
# an un-migrated DB, crashes, and (with the restart policy) the container
136+
# restarts. That restart SIGKILLs any `docker exec` running inside it -> the
137+
# migration dies with exit 137. So we stop the app and run the migration as a
138+
# one-off compose container (overridden entrypoint = migrate only, no webapp,
139+
# no restart policy), using the same compose files/env as the deploy step.
148140
# ---------------------------------------------------------------
149-
( while true; do \
150-
echo "[stats $(date +%H:%M:%S)] $(docker stats --no-stream --format '{{.Name}} mem={{.MemUsage}} ({{.MemPerc}}) cpu={{.CPUPerc}}' $DNAME $DDBNAME 2>/dev/null | tr '\n' ' ')"; \
151-
sleep 2; \
152-
done ) &
153-
STATS_PID=$!
154141
155-
echo "Running DSpace database migration..."
156-
# Do NOT let 'set -e' abort before we print diagnostics: capture the rc.
157-
MIGRATE_RC=0
158-
docker exec $DNAME /bin/bash -c "cd /dspace/bin && ./dspace database migrate ignored" || MIGRATE_RC=$?
159-
160-
# stop the stats sampler
161-
kill $STATS_PID 2>/dev/null || true
142+
echo "Stopping app container so its restart loop can't kill the migration..."
143+
docker stop $DNAME || true
162144
163-
echo "===== DEBUG: migrate exit code = $MIGRATE_RC ====="
145+
echo "Running one-off DB migration via docker compose..."
146+
docker compose --env-file $ENVFILE -p dspace-${{inputs.INSTANCE}} \
147+
-f docker/docker-compose.yml -f docker/docker-compose-rest.yml \
148+
-f $OVERLAY/docker-compose-rest.yml -f $OVERLAY/docker-compose.yml \
149+
run --rm --no-deps --entrypoint /bin/bash dspace \
150+
-c "while (!</dev/tcp/dspacedb/5432) >/dev/null 2>&1; do sleep 1; done; cd /dspace/bin && ./dspace database migrate ignored"
164151
165-
# ---------------------------------------------------------------
166-
# DEBUG: post-mortem state AFTER migration (runs even on failure)
167-
# ---------------------------------------------------------------
168-
echo "===== DEBUG: container state / OOM flag AFTER ====="
169-
docker inspect $DNAME --format 'status={{.State.Status}} exit={{.State.ExitCode}} OOMKilled={{.State.OOMKilled}} restarts={{.RestartCount}}' || true
170-
echo "===== DEBUG: host memory (free -h) AFTER ====="
171-
free -h || true
172-
echo "===== DEBUG: kernel OOM-killer evidence (dmesg) ====="
173-
(sudo dmesg -T 2>/dev/null || dmesg -T 2>/dev/null || true) | grep -iE 'oom|killed process|out of memory|java' | tail -30 || true
174-
echo "===== DEBUG: last 60 lines of dspace-cli.log ====="
175-
docker exec $DNAME /bin/bash -c "tail -n 60 /dspace/log/dspace-cli.log" || true
176-
177-
if [ "$MIGRATE_RC" -ne 0 ]; then
178-
echo "Migration failed with exit code $MIGRATE_RC (137 = 128+9 = SIGKILL, typically OOM)."
179-
exit $MIGRATE_RC
180-
fi
152+
echo "Migration done. Starting app container (entrypoint migrate is now a no-op)..."
153+
docker start $DNAME
181154
182155
echo "Waiting for DSpace to be ready..."
183156
sleep 10

0 commit comments

Comments
 (0)