Skip to content

Commit 6cf31b4

Browse files
jmclaude
andcommitted
feat(dev): auto-recognise backend image + FLAVOR (upstream|clarin)
dev.backend.sh no longer hardcodes the backend image. It reads the image tags from the repo's own compose files (docker-compose-rest.yml / docker-compose-ci.yml) and selects a set by FLAVOR, then after boot recognises the running backend (image / version / flavor + a browse-definitions canary) and warns loudly on a DSpace-7.5 mismatch. - FLAVOR=upstream (default): public DSpace 7.6.x demo + db.entities.yml - FLAVOR=clarin: version-correct dataquest/dspace:dspace-7_x-test (read from docker-compose-ci.yml, NOT the rest.yml dtq-dev-7.5 default) + CLARIN content via new docker/db.clarin.yml (CI's loadsql + dataquest dump) - UI_PORT now single-sources both UI_URL and REST_CORS_ALLOWED_ORIGINS so the FE port and BE CORS can't drift - AGENTS.md: new "Backend image - recognised automatically" section + updated Gotcha #6 (version match is now auto-checked) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent d969edb commit 6cf31b4

3 files changed

Lines changed: 169 additions & 28 deletions

File tree

AGENTS.md

Lines changed: 48 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -41,16 +41,22 @@ Builds on PR #1289, which makes `docker/docker-compose-rest.yml` drivable from o
4141
nvm use # or: see Prerequisites for a portable Node 18
4242
yarn install
4343

44-
# 1) BACKEND: DSpace 7.6.5 + demo content at http://127.0.0.1:8087/server (~2-4 min first run)
45-
build-scripts/run/dev.backend.sh # add 'fresh' to wipe volumes + reload from scratch
44+
# 1) BACKEND: DSpace 7.6.x at http://127.0.0.1:8087/server (~2-4 min first run). The image set is
45+
# recognised automatically (read from the compose files) and the script prints + validates the
46+
# running version/flavor after boot. Two flavors:
47+
build-scripts/run/dev.backend.sh # upstream public demo (default): right version, public content
48+
FLAVOR=clarin build-scripts/run/dev.backend.sh fresh # the CORRECT CLARIN backend (dataquest) + CLARIN content
49+
# ('fresh' wipes volumes — required when switching flavor, since -loadsql only loads an empty DB)
4650

4751
# 2) FRONTEND: live-reload dev server on http://localhost:4000
4852
yarn start:dev:local
4953
```
5054

51-
`dev.backend.sh` brings the backend up (correct 7.6.5 images, IPv4 host, CORS, demo dataset)
52-
and reindexes Solr; `start:dev:local` is `ng serve` with the right `DSPACE_REST_*` env baked in.
53-
Stop/wipe the backend with `docker compose -f docker/docker-compose-rest.yml -f docker/db.entities.yml down -v`.
55+
`dev.backend.sh` brings the backend up (IPv4 host, CORS derived from `UI_PORT`, sample dataset),
56+
reindexes Solr, then **recognises the running backend** and warns if its version/flavor won't match
57+
this FE; `start:dev:local` is `ng serve` with the right `DSPACE_REST_*` env baked in.
58+
Stop/wipe with `docker compose -f docker/docker-compose-rest.yml -f docker/db.entities.yml down -v`
59+
(swap in `-f docker/db.clarin.yml` for the CLARIN flavor).
5460

5561
Everything below is the manual/explained version of those two commands (for customizing the
5662
instance, image set, or running the FE in Docker). The FE maps env vars onto `config/config.yml`
@@ -63,6 +69,37 @@ instance, image set, or running the FE in Docker). The FE maps env vars onto `co
6369
`docker/docker-compose-rest.yml` fully defines the network, so the BE comes up from that file
6470
(plus `db.entities.yml` for demo data) — no `docker-compose.yml` (that's the FE container).
6571

72+
### Backend image — recognised automatically (don't hardcode it)
73+
74+
`dev.backend.sh` does **not** hard-code the backend image. It **reads the image tags from the repo's
75+
own compose files** (the source of truth), selects a set by `FLAVOR`, and after boot recognises the
76+
running backend — printing its image, version, flavor, and a browse-definitions canary (the DSpace-7.5
77+
mismatch trap). The image the compose file declares as its *default* is deliberately **not** the one to
78+
run as-is:
79+
80+
| Source (in the code) | REST image | Flavor | Version |
81+
|---|---|---|---|
82+
| `docker/docker-compose-rest.yml:70` (compose default) | `dataquest/dspace:dtq-dev-7.5` | CLARIN | **7.5 — breaks browse-defs vs this 7.6.x FE** |
83+
| `docker/docker-compose-ci.yml:38` (what CI validates) | `dataquest/dspace:dspace-7_x-test` | CLARIN | 7.6.x ✅ |
84+
| `dev.backend.sh` `FLAVOR=upstream` (default) | `dspace/dspace:dspace-7_x` | upstream demo | 7.6.x ✅ |
85+
86+
So `FLAVOR=clarin` reads the **CI** default (version-correct CLARIN), *not* the rest.yml default;
87+
`FLAVOR=upstream` (default) keeps the instant public-demo path. The post-boot recogniser prints:
88+
89+
```
90+
Backend recognised:
91+
image : dspace/dspace:dspace-7_x
92+
version : DSpace 7.6.7-SNAPSHOT (this FE is 7.6.5)
93+
flavor : upstream demo (CLARIN endpoints/content ABSENT)
94+
browse-definitions canary : HTTP 200 (OK)
95+
note: for the CORRECT CLARIN backend (CLARIN endpoints + content) run:
96+
FLAVOR=clarin build-scripts/run/dev.backend.sh fresh
97+
```
98+
99+
(`FLAVOR=clarin` uses `docker/db.clarin.yml`, the CLARIN counterpart of `db.entities.yml`: a `-loadsql`
100+
Postgres fed the dataquest CLARIN test dump — same image + dump that CI validates. First run pulls
101+
~1-2 GB of dataquest images.)
102+
66103
`docker/.env.local` for this recipe:
67104

68105
```ini
@@ -171,10 +208,12 @@ errors are normal: `favicon.ico` 404, Matomo refused, `google.analytics.key` 404
171208
**both** `DSPACE_REST_HOST` and `REST_URL` on `127.0.0.1`. (`curl` hides this — it falls back
172209
to IPv4.)
173210

174-
6. **Match the BE to the FE version.** The FE reports **7.6.5**, so use a **7.6.5** BE
175-
(`dspace/dspace:dspace-7_x` or `dataquest/dspace:dspace-7_x`). The `dataquest/dspace:dtq-dev-7.5`
176-
image is **DSpace 7.5** and causes `An error occurred while retrieving the browse definitions`
177-
in the console.
211+
6. **Match the BE to the FE version — now auto-checked.** The FE is **7.6.x**, so use a **7.6.x** BE.
212+
The `dataquest/dspace:dtq-dev-7.5` image (the compose *default*, see "Backend image — recognised
213+
automatically") is **DSpace 7.5** and causes `An error occurred while retrieving the browse
214+
definitions` in the console. `dev.backend.sh` now handles this for you: it picks a 7.6.x image by
215+
`FLAVOR` and runs a browse-definitions canary after boot, warning loudly if a 7.5-style BE is
216+
detected. For CLARIN fidelity use `FLAVOR=clarin` (the version-correct `dataquest/dspace:dspace-7_x-test`).
178217

179218
7. **Demo data ≠ empty repo, and Solr needs reindexing.** A fresh DB shows an empty homepage —
180219
that's expected, not a bug. Layer `db.entities.yml` (Option A) for sample content, then run

build-scripts/run/dev.backend.sh

Lines changed: 87 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -2,21 +2,35 @@
22
#
33
# One-command local-dev BACKEND for working on the native frontend.
44
#
5-
# Brings up a DSpace 7.6.5 backend (matching this FE's version) in Docker, loaded with the
6-
# official demo entities dataset, reachable at http://127.0.0.1:8087/server, then indexes Solr
7-
# so browse/search/"What's New" are populated. After it prints "Backend ready", start the FE:
5+
# Brings up a DSpace backend in Docker (matching this FE), loaded with a sample dataset, reachable at
6+
# http://127.0.0.1:8087/server, then indexes Solr so browse/search/"What's New" populate. After it
7+
# prints "Backend ready", start the FE:
88
#
99
# yarn start:dev:local # ng serve, live-reload, http://localhost:4000 (needs Node 18 — see .nvmrc)
1010
#
11+
# The backend IMAGE is recognised AUTOMATICALLY rather than hardcoded here:
12+
# * image tags are read from the repo's own compose files (the source of truth, so they can't go
13+
# stale against a copy in this script), and
14+
# * after boot the script recognises the running backend's version + flavor and warns if it does
15+
# not match this FE (the DSpace-7.5 "browse definitions" trap).
16+
#
17+
# FLAVOR selects WHICH backend (override: FLAVOR=clarin ...):
18+
# upstream (default) public DSpace demo — DSpace 7.6.x + the official demo dataset (db.entities.yml).
19+
# Right VERSION for this FE and zero dataquest pull, but UPSTREAM flavor: CLARIN-specific
20+
# endpoints/config/content are absent (some FE calls 404 — harmless to render).
21+
# clarin the version-correct CLARIN/dataquest backend — the image DSpace CI validates
22+
# (dataquest/dspace:dspace-7_x-test, read from docker/docker-compose-ci.yml) + the CLARIN
23+
# test dataset (docker/db.clarin.yml). This is the "correct backend" for CLARIN feature
24+
# work. NOTE: first run pulls ~1-2 GB of dataquest images.
25+
#
1126
# Usage:
12-
# build-scripts/run/dev.backend.sh # up (reuses existing containers/data)
13-
# build-scripts/run/dev.backend.sh fresh # wipe DB/Solr volumes first, then up (clean slate)
27+
# build-scripts/run/dev.backend.sh # up (reuses existing containers/data)
28+
# build-scripts/run/dev.backend.sh fresh # wipe DB/Solr volumes first (REQUIRED to switch FLAVOR)
29+
# FLAVOR=clarin build-scripts/run/dev.backend.sh fresh
1430
#
1531
# Notes:
1632
# - 127.0.0.1 (not localhost): Node resolves localhost to IPv6 ::1, but the BE binds IPv4 only.
17-
# - Demo data is UPSTREAM DSpace (not CLARIN), so some CLARIN-specific FE calls 404 — harmless.
18-
# For CLARIN content, point DSPACE_REST_IMAGE at dataquest/dspace:dspace-7_x and restore a
19-
# dataquest/LINDAT DB dump instead of using db.entities.yml.
33+
# - -loadsql images import only into an EMPTY pgdata volume, so always `fresh` when switching FLAVOR.
2034
#
2135
set -uo pipefail
2236
cd "$(dirname "$0")/../.." || exit 1
@@ -26,25 +40,77 @@ export COMPOSE_PROJECT_NAME="dspace-${INSTANCE}"
2640
export DSPACE_HOST=127.0.0.1
2741
export DSPACE_REST_NAMESPACE=/server
2842
export REST_URL="http://127.0.0.1:808${INSTANCE}/server"
29-
export UI_URL="http://localhost:4000"
43+
# FE dev-server port; CORS is derived from it so the two can never drift (set UI_PORT to change both).
44+
export UI_PORT="${UI_PORT:-4000}"
45+
export UI_URL="http://localhost:${UI_PORT}"
3046
export HOST_IP=127.0.0.1
3147
export DSPACE_SUBNET_PREFIX="10.10${INSTANCE}"
32-
export REST_CORS_ALLOWED_ORIGINS="http://localhost:4000,http://127.0.0.1:4000"
33-
# DSpace 7.6.5 to match the FE; upstream images + the demo entities dataset (db.entities.yml).
34-
export DSPACE_REST_IMAGE=dspace/dspace:dspace-7_x
35-
export DSPACE_DB_IMAGE=dspace/dspace-postgres-pgcrypto:dspace-7_x
36-
export DSPACE_SOLR_IMAGE=dspace/dspace-solr:dspace-7_x
37-
export DOCKER_REGISTRY=docker.io DOCKER_OWNER=dspace DSPACE_VER=dspace-7_x
48+
export REST_CORS_ALLOWED_ORIGINS="http://localhost:${UI_PORT},http://127.0.0.1:${UI_PORT}"
49+
export DSPACE_VER="${DSPACE_VER:-dspace-7_x}"
50+
export DOCKER_REGISTRY="${DOCKER_REGISTRY:-docker.io}"
3851

39-
COMPOSE=(docker compose -f docker/docker-compose-rest.yml -f docker/db.entities.yml)
52+
REST_YML=docker/docker-compose-rest.yml
53+
CI_YML=docker/docker-compose-ci.yml
4054
REST="http://127.0.0.1:808${INSTANCE}/server/api"
4155

56+
# Read the default value of a compose `${VAR:-default}` straight from the file (source of truth).
57+
compose_default() { grep -oE "\\\$\\{$1:-[^}]+\\}" "$2" | head -1 | sed -E 's/^.*:-//; s/}$//'; }
58+
59+
# --- recognise WHICH backend image to use (by FLAVOR); tags come FROM THE REPO, not hardcoded here ---
60+
FLAVOR="${FLAVOR:-upstream}"
61+
if [ "$FLAVOR" = "clarin" ]; then
62+
# Version-correct CLARIN REST image = the default DSpace CI runs against (docker-compose-ci.yml).
63+
# Deliberately NOT the rest.yml default (dataquest/dspace:dtq-dev-7.5 is DSpace 7.5, which breaks
64+
# browse-definitions against this 7.6.x FE — see the recognise step below and AGENTS.md Gotcha #6).
65+
export DSPACE_REST_IMAGE="${DSPACE_REST_IMAGE:-$(compose_default DSPACE_CI_IMAGE "$CI_YML")}"
66+
export DSPACE_SOLR_IMAGE="${DSPACE_SOLR_IMAGE:-$(compose_default DSPACE_SOLR_IMAGE "$REST_YML")}"
67+
export DOCKER_OWNER="${DOCKER_OWNER:-dataquest}"
68+
COMPOSE=(docker compose -f "$REST_YML" -f docker/db.clarin.yml)
69+
else
70+
# Upstream public demo: right version family (7.6.x) + the official demo dataset (db.entities.yml).
71+
export DSPACE_REST_IMAGE="${DSPACE_REST_IMAGE:-dspace/dspace:dspace-7_x}"
72+
export DSPACE_SOLR_IMAGE="${DSPACE_SOLR_IMAGE:-dspace/dspace-solr:dspace-7_x}"
73+
export DOCKER_OWNER="${DOCKER_OWNER:-dspace}"
74+
COMPOSE=(docker compose -f "$REST_YML" -f docker/db.entities.yml)
75+
fi
76+
echo ">> FLAVOR=${FLAVOR} DSPACE_REST_IMAGE=${DSPACE_REST_IMAGE}"
77+
78+
# Recognise the RUNNING backend (whoever/whatever started it) and warn if it won't match this FE.
79+
recognise_backend() {
80+
local img ver fe browse flavor compat
81+
img="$(docker inspect "dspace${INSTANCE}" --format '{{.Config.Image}}' 2>/dev/null || true)"
82+
ver="$(curl -s "$REST" | sed -nE 's/.*"dspaceVersion" *: *"([^"]+)".*/\1/p' | head -1)"
83+
fe="$(sed -nE 's/.*"version": *"([0-9][^"]*)".*/\1/p' package.json | head -1)"
84+
browse="$(curl -s -o /dev/null -w '%{http_code}' "$REST/discover/browses" 2>/dev/null)"
85+
case "$img" in
86+
dataquest/*) flavor="CLARIN (dataquest)";;
87+
dspace/*) flavor="upstream demo (CLARIN endpoints/content ABSENT)";;
88+
*) flavor="unknown (${img:-?})";;
89+
esac
90+
[ "$browse" = "200" ] && compat="OK" || compat="MISMATCH — looks like a DSpace 7.5 backend"
91+
echo "------------------------------------------------------------------"
92+
echo " Backend recognised:"
93+
echo " image : ${img:-?}"
94+
echo " version : ${ver:-?} (this FE is ${fe:-7.6.x})"
95+
echo " flavor : ${flavor}"
96+
echo " browse-definitions canary : HTTP ${browse} (${compat})"
97+
case "$ver" in
98+
'DSpace 7.6'*|'DSpace 7.7'*|'DSpace 8'*) ;;
99+
*) echo " !! BE version may not match the FE; if the canary is not 200, use a 7.6.x image." >&2;;
100+
esac
101+
if [ "$FLAVOR" != "clarin" ] && [ "${img#dspace/}" != "$img" ]; then
102+
echo " note: for the CORRECT CLARIN backend (CLARIN endpoints + content) run:"
103+
echo " FLAVOR=clarin $0 fresh"
104+
fi
105+
echo "------------------------------------------------------------------"
106+
}
107+
42108
if [ "${1:-}" = "fresh" ]; then
43109
echo ">> wiping previous dev backend (down -v)"
44110
"${COMPOSE[@]}" down -v --remove-orphans || true
45111
fi
46112

47-
echo ">> starting backend: DSpace 7.6.5 + demo entities (project ${COMPOSE_PROJECT_NAME})"
113+
echo ">> starting backend (project ${COMPOSE_PROJECT_NAME}, flavor ${FLAVOR})"
48114
if ! "${COMPOSE[@]}" up -d; then
49115
echo "!! 'up' failed. If it's a Postgres version/volume mismatch, run: $0 fresh" >&2
50116
exit 1
@@ -65,12 +131,14 @@ echo ">> indexing Solr discovery (so browse/search/What's New populate)"
65131
MSYS_NO_PATHCONV=1 docker exec "dspace${INSTANCE}" /dspace/bin/dspace index-discovery -b \
66132
|| echo " (reindex failed — rerun: MSYS_NO_PATHCONV=1 docker exec dspace${INSTANCE} /dspace/bin/dspace index-discovery -b)"
67133

134+
recognise_backend
135+
68136
cat <<MSG
69137
70138
==================================================================
71-
Backend ready: ${REST%/api} (DSpace 7.6.5, demo content)
139+
Backend ready: ${REST%/api} (flavor: ${FLAVOR})
72140
Start the FE : yarn start:dev:local (Node 18 — see .nvmrc)
73-
Then open : http://localhost:4000/
141+
Then open : ${UI_URL}/
74142
Stop / wipe : ${COMPOSE[*]} down -v
75143
==================================================================
76144
MSG

docker/db.clarin.yml

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
#
2+
# CLARIN/dataquest content layer for LOCAL DEV.
3+
# Used by build-scripts/run/dev.backend.sh when FLAVOR=clarin.
4+
#
5+
# This is the CLARIN counterpart of docker/db.entities.yml: instead of the upstream demo dataset it
6+
# loads the dataquest CLARIN test database dump, and it skips the upstream Entities-specific
7+
# item-submission tweaks. The image + dump are the same ones DSpace CI validates
8+
# (see docker/docker-compose-ci.yml: DSPACE_CI_IMAGE / LOADSQL).
9+
#
10+
# Layer it on top of docker-compose-rest.yml:
11+
# docker compose -f docker/docker-compose-rest.yml -f docker/db.clarin.yml up -d
12+
#
13+
# IMPORTANT: a -loadsql image only imports into an EMPTY pgdata volume, so switch flavors clean:
14+
# FLAVOR=clarin build-scripts/run/dev.backend.sh fresh
15+
#
16+
services:
17+
dspacedb:
18+
# Upstream "loadsql" loader image (already pulled for the demo path). The loader flavor is
19+
# irrelevant — the CONTENT is CLARIN, fetched from ${LOADSQL} on first boot.
20+
image: ${DOCKER_REGISTRY:-docker.io}/dspace/dspace-postgres-pgcrypto:${DSPACE_VER:-dspace-7_x}-loadsql
21+
environment:
22+
# dataquest CLARIN test DB dump. Keep in sync with docker/docker-compose-ci.yml.
23+
- LOADSQL=${LOADSQL:-https://github.com/dataquest-dev/DSpace/releases/download/data/dspace-test-database-dump_29.1.2024.sql}
24+
dspace:
25+
### OVERRIDE the default 'entrypoint' from docker-compose-rest.yml ###
26+
# 1. wait until Postgres is reachable, 2. migrate (ignored = tolerate out-of-order CLARIN
27+
# migrations), 3. start Tomcat. dev.backend.sh runs `index-discovery -b` once the API is up.
28+
entrypoint:
29+
- /bin/bash
30+
- '-c'
31+
- |
32+
while (!</dev/tcp/dspacedb/5432) > /dev/null 2>&1; do sleep 1; done;
33+
/dspace/bin/dspace database migrate ignored
34+
catalina.sh run

0 commit comments

Comments
 (0)