Give every service both recovery paths, and build the image instead of pulling

Two problems surfaced the first time this compose file was run end to end.

1. The stack did not start at all, and it had nothing to do with healthchecks.

docker-compose.yml mounts ./dspace/config over the config inside the image, so
the image has to be built from this tree. It is not enough to pull a matching
version: this branch carries code no published image contains - for instance
dspace/config/ehcache.xml references
org.dspace.external.provider.orcid.xml.CacheLogger, which is in dspace-api here
but in neither upstream 7.6.5 nor 7.6.8. A pulled image dies with
"Error parsing XML configuration at file:/dspace/config/ehcache.xml", and before
that with "Could not resolve placeholder 'pubmed.apiKey'".

`docker compose up` builds a missing image on its own, so a clean machine never
sees this. The trap is a stale image from an earlier pull silently shadowing the
build, which makes the failure look like a config problem. DSPACE_VER is now
pinned to the pom.xml version instead of the floating dspace-7_x tag so a newer
upstream image cannot take its place, and the file says so in a comment.

2. Only the backend and the frontend could actually recover.

dspacedb and dspacesolr had neither a restart policy nor the autoheal label, so
"the database goes down and comes back on its own" simply did not happen - it
stayed down. Every service now carries both, because they cover different
failures: the restart policy handles a dead process and an exited container,
the sidecar handles a container that still runs while its healthcheck fails.

Verified by breaking each service in a different way:

  dspacedb   pg_ctl stop -m immediate -> container exited, RestartCount 0 -> 1,
             back to healthy on its own
  dspacesolr search core unloaded -> failing streak 1..5 -> unhealthy ->
             restarted by the sidecar
  dspace     chaos flag -> unhealthy -> restarted
  dspace-angular  ng serve killed -> container exited -> restarted

One correction to an earlier claim in this file: a restart does NOT repair an
unloaded Solr core. precreate-core only checks whether the core directory
exists, so it logs "Core search already exists" and leaves the core
unregistered. The comment now says that, and it is precisely the case the
sidecar's restart budget exists for - it gives up and asks for a human instead
of looping.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Author: MatusBeke

docker-compose.yml

@@ -1,4 +1,24 @@
 version: '3.7'
+# IMAGE VERSION - build it, do not pull it
+#
+# The dspace image MUST be built from this repository:
+#
+#     docker compose build dspace
+#
+# This branch carries customisations that no published image contains. For example
+# dspace/config/ehcache.xml references org.dspace.external.provider.orcid.xml.CacheLogger,
+# which exists in dspace-api here but in neither upstream 7.6.5 nor 7.6.8. Since this
+# compose file also mounts ./dspace/config over the image config, a pulled image gives
+# you a newer/other webapp reading this tree config, and the DSpace kernel dies with
+# "Error parsing XML configuration at file:/dspace/config/ehcache.xml".
+#
+# `docker compose up` builds a missing image on its own, so a clean machine is fine.
+# The trap is a STALE image left over from an earlier pull: it shadows the build and
+# the failure looks unrelated to images. `docker compose build` is the fix.
+#
+# DSPACE_VER is pinned to the pom.xml version rather than the floating dspace-7_x tag
+# so the local build cannot be silently shadowed by a newer upstream one. Bump it
+# together with pom.xml.
 networks:
   dspacenet:
     ipam:
@@ -46,7 +66,7 @@ services:
       # HeapDumpOnOutOfMemoryError writes the dump to the mounted log volume so the
       # cause survives the restart. Without it the restart destroys the evidence.
       JAVA_OPTS: '-Xmx2000m -XX:+ExitOnOutOfMemoryError -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/dspace/log'
-    image: "${DOCKER_OWNER:-dspace}/dspace:${DSPACE_VER:-dspace-7_x-test}"
+    image: "${DOCKER_OWNER:-dspace}/dspace:${DSPACE_VER:-dspace-7.6.5}-test"
     build:
       context: .
       dockerfile: Dockerfile.test
@@ -137,8 +157,16 @@ services:
   # DSpace PostgreSQL database container
   dspacedb:
     container_name: dspacedb
+    # Two independent recovery paths, both needed:
+    #   restart policy -> the postmaster died and the container exited
+    #   autoheal label -> the container still runs but pg_isready keeps failing
+    # Restarting a database is not free, which is why the sidecar enforces a restart
+    # budget (3 per hour by default) and then stops and asks for a human.
+    restart: unless-stopped
+    labels:
+      autoheal: "true"
     # Uses a custom Postgres image with pgcrypto installed
-    image: "${DOCKER_OWNER:-dspace}/dspace-postgres-pgcrypto:${DSPACE_VER:-dspace-7_x}"
+    image: "${DOCKER_OWNER:-dspace}/dspace-postgres-pgcrypto:${DSPACE_VER:-dspace-7.6.5}"
     build:
       # Must build out of subdirectory to have access to install script for pgcrypto
       context: ./dspace/src/main/docker/dspace-postgres-pgcrypto/
@@ -167,7 +195,19 @@ services:
   # DSpace Solr container
   dspacesolr:
     container_name: dspacesolr
-    image: "${DOCKER_OWNER:-dspace}/dspace-solr:${DSPACE_VER:-dspace-7_x}"
+    # Two independent recovery paths, both needed:
+    #   restart policy -> the Solr process died and the container exited
+    #   autoheal label -> the container still runs but the search core is gone
+    #
+    # Note what a restart does NOT fix: precreate-core only checks whether the core
+    # DIRECTORY exists, so once a core has been unloaded (its core.properties is gone
+    # but the directory stays) a restart logs "Core search already exists" and moves on,
+    # leaving the core unregistered. That is intentional on the sidecar side - it burns
+    # its restart budget, gives up, and asks for a human, which beats looping forever.
+    restart: unless-stopped
+    labels:
+      autoheal: "true"
+    image: "${DOCKER_OWNER:-dspace}/dspace-solr:${DSPACE_VER:-dspace-7.6.5}"
     build:
       context: ./dspace/src/main/docker/dspace-solr/
       # Provide path to Solr configs necessary to build Docker image

dspace/src/main/docker-compose/docker-compose-angular.yml

@@ -15,6 +15,7 @@ networks:
 services:
   dspace-angular:
     container_name: dspace-angular
+    restart: unless-stopped
     depends_on:
       dspace:
         condition: service_healthy
@@ -27,7 +28,7 @@ services:
       DSPACE_REST_HOST: localhost
       DSPACE_REST_PORT: 8080
       DSPACE_REST_NAMESPACE: /server
-    image: dspace/dspace-angular:dspace-7_x
+    image: "${DOCKER_OWNER:-dspace}/dspace-angular:${DSPACE_VER:-dspace-7.6.5}"
     ports:
     - published: 4000
       target: 4000