Merge compose files from multiple repos

[DocKDE]

I have the following issue: I am deploying Immich and the project publishes their own compose file as a release artifact. I'd like to use this as a source of truth, pull it in and merge it with my own configs.

I tried using include in my compose file:

include:
  - https://github.com/immich-app/immich.git#v3.0.3:docker/docker-compose.yml

but I kept getting errors that the upstream repository could not be found.
I am aware that I can define a repository_url in my .doco-cd.yaml which I could point at the immich upstream, but as I understand it, this will use the compose file in that repo and ignore any overlays or custom compose files I have in my own application deployment repo.
I could periodically pull the upstream compose file and incorporate it into my own repo, but that's rather tedious and I'd rather automate this.

Is there a doco-cd way to do this kind of thing and I missed it? It seems like something that should be possible, somehow.

[kimdre]

This should work without problems. It works on my test setup. If not, check if you use a Git access token that does not have the required permissions for this.

include:
  - https://github.com/immich-app/immich.git#v3.0.3:docker/docker-compose.yml

However the compose file provided by immich wants a .env file, which is not possible to create in this context, because the cloned immich repository sits in a different cache drectory than your own deployment repository so it's not possible to share a .env file between the two. When doco-cd loads the compose config, it will fail while trying to do the variable interpolation using the missing .env file.

[kimdre]

What you can do however is to add the immich repository as a submodule to your deployment repository and continue from there using include with a relative path to the submodule and the .env file in your repo root.

[DocKDE]

I did try various access tokens. Currently, I am using a fine-grained one with Public Repositories (read-only) selected. Since I only want to access public repos, that should be all that is necessary or am I missing something?

I don't quite understand the issue with the .env file. Generally, the include directive will fetch a compose file (usually from the local file system) and merge it with the one that calls it, which itself may define .env files which can sit next to it. It was my understanding that specifying a git repo as include target would behave the same, but fetch the compose file from said git repo. But apparently this is not the case?

[kimdre]

You can try to enable the debug logs and check what doco-cd is trying to do.

When a Git repository is specified in include, doco-cd will clone the repository to a cache directory and then load the compose file specified in the subdirectory.

I just tested it with project_directory like you did below and that seems to be the trick when the included compose file wants custom configuration files. 😄 Without that, doco-cd tries to load everything from the path of the cached/cloned immich repository instead of your own repository as documented in the compose reference docs.

I will document this in the doco-cd wiki -> https://doco.cd/latest/Advanced/Tips-and-Tricks/#using-include-with-remote-compose-files

[DocKDE]

Yes, this is as documented for the include directive. The same is true for .env files, by default the working directory is that of the included compose file and in the case of a git repo this lives in some local cache dir. It gets tricky when you want to use files from your local repo and the remote git repo, because with an include of a local file you still know all the relevant paths to work around this. When including a remote git repo this is no longer true (at least not necessarily), so it's only practical to use config and .env files from one of the sources.

[kimdre]

In that case you need to use the immich repository as a submodule in your deployment repository.

git submodule add --depth 1 -b v3.0.3 https://github.com/immich-app/immich.git

Then you can reference everything with relative paths

include:
  - path: immich/docker/docker-compose.yml
    env_files:
      - ../.env

[DocKDE]

I just tried this:

include:
  - path: https://github.com/immich-app/immich.git#${IMMICH_VERSION}:docker/docker-compose.yml
    project_directory: .

And it seems to work as intended. Any config files need to be supplied by myself in this case, but that is acceptable, I think.