linderman

Docker compose deployment of various small, internal apps for Lehigh's Library Technology team.

• Docker compose is the container orchestrator
• Traefik handles TLS, routing to apps, LDAP authentication
• Each app is a service in the docker compose YAML, and served as a route/path under our main domain
• GitHub Actions + self hosted runner + rollout service handle code deploys. More info in continuous deployment section
• This stack is deployed into SET managed VMs in Lehigh's data center

Requirements

• Docker 24.0+
• Docker Compose 2.x
• git

Local development setup

First, clone your app(s)

git clone git@github.com:lehigh-university-libraries/folio-offline-shelf-reading

Next, clone this repo, which is configured to run all apps Library Technology deploys using linderman. Then run the script that generates a self-signed cert.

Warning

Ensure you have mkcert 1.4+ installed

git clone git@github.com:lehigh-university-libraries/linderman
cd linderman
./scripts/maintenance/generate-certs.sh
./scripts/maintenance/create-secrets.sh

Start the services

docker compose up -d

You should now be able to view the apps at e.g. https://localhost/shelf-reading

You should be able to make edits to your app's code, which should be git clone'd into the same directory linderman was cloned into

.
├── folio-offline-shelf-reading
├── folio-shelving-order
├── linderman

If you need to make edits to the dockerfile on a specific app (e.g. installing a new pip dependency), you can build the docker image for your app to get the dependency installed with. e.g.

cd /path/to/linderman
docker compose up --build folio-shelving-order -d

Adding a New Service

To add a new service to linderman, follow these steps:

1. Define Traefik router and service in conf/traefik/config.tmpl
   • Add a router that matches the path prefix for your service
   • Add a corresponding service pointing to your docker container

http:
  routers:
    my-app:
      rule: "PathPrefix(`/my-app`)"
      service: my-app
  services:
    my-app:
      loadBalancer:
        servers:
          - url: "http://my-app:8000"

2. Create secrets directory (if needed)
   • If your service requires secrets (API keys, credentials, etc.), create a directory in secrets/ named after your docker compose service name
   • Add the necessary secret files to this directory
   • these secrets will get created automatically for you on the host VMs but you'll need to populate them with their contents

mkdir -p secrets/my-app
echo "secret-value" > secrets/my-app/api-key

3. Add service definition to docker-compose.yaml
   • Define your service with the appropriate image, volumes, environment variables, etc.
   • Ensure the service name matches what you referenced in the Traefik configuration

services:
  my-app:
    image: my-org/my-app:latest
    environment:
      SCRIPT_NAME: /my-app
    volumes:
      - ./secrets/my-app/api-key:/app/api-key:ro

4. Configure app for path prefix compatibility

   • Your application must be compatible with running under a Traefik path prefix (e.g. /shelf-reading)
   • For Python/Flask/Gunicorn apps, set the SCRIPT_NAME environment variable on the container to match your path prefix
   • Example: SCRIPT_NAME=/my-app in your docker-compose service definition (shown above in step 2)

5. Ensure rollout.sh knows how to rollout your app

• ./scripts/maintenance/rollout.sh needs to know which docker compose service needs restarted when you app deploys to linderman. This is done by specifying the docker tag environment variable, docker compose service name, and git repo that will trigger the deploy. You can use the other if/elif statements as an example how to add this

Continuous Deployment

This repo, as well as each app linderman hosts, references a reusable GitHub Action linderman-deploy.yaml to deploy changes made in GitHub into Lehigh's infrastructure.

That shared action leverages linderman's self-hosted GitHub Action Runner, defined in docker-compose.libapps-test.yaml to trigger a rollout when pushes are made to a branch. That GitHub Action runner was added to the lehigh-university-libraries GitHub org so any repo in our org can leverage the self hosted runner.

We need a self-hosted runner since the linderman services are protected via a firewall to on-campus only. The rollout workflow is basically:

• slack alert message when rollout starts
• if an app is being deployed, run docker pull for the app's docker tag
• else if this repo/linderman is what's being deployed, run git pull on the filesystem
• run docker compose up -d to get the changes running
• slack alert message when rollout ends (pass or fail status)

Each app can define in their GitHub Action when to deploy to test or prod. This repo deploys to test whenever a branch is pushed to this repo, and when the branch is merged into main that is deployed to test and then prod.

Rollout Service

The logic performed during the rollout can be seen in rollout.sh. That script is executed by the GitHub Action using OIDC/JWT auth on the rollout docker service. So triggering a rollout is basically just a cURL call from a GitHub Action.

Manual deployment

If ever needed, you can manually deploy linderman like so

ssh apps-test.lib.lehigh.edu
cd /opt/linderman
sudo git checkout main
sudo git pull origin main
sudo docker compose pull
sudo systemctl restart linderman

Same steps for production, except start with ssh apps-prod.lib.lehigh.edu

Service/App Authentication

Any service running in linderman can leverage LDAP authentication by adding the traefik middleware ldap-valid-user to its traefik router in config/traefik/config.tmpl. This will force LDAP authentication before the application can be loaded in the web browser, and once authenticated will forward the username via the HTTP header X-Remote-User to the backend service.

The LDAP traefik middleware is maintained at https://github.com/lehigh-university-libraries/ldapAuth and updates can be pulled into linderman via

cd path/to/linderman
git subtree pull --prefix conf/traefik/plugins/ldapAuth https://github.com/lehigh-university-libraries/ldapAuth main --squash

App-specific allowed users

If an app is restricted to certain users, for privacy/security reasons, instead of hardcoding the list of users in version control, the allowed users are set in ./conf/traefik/ldap.yml in the allowedUsers list:

    ldap-shelf-reading:
      plugin:
        ldapAuth:
          enabled: true
          attribute: "uid"
          baseDN: "dc=lehigh,dc=edu"
          logLevel: INFO
          serverList:
            - Url: "ldaps://nis.cc.lehigh.edu"
              Port: 636
              Weight: 10
          allowedUsers:
            - bob
            - alice
            - terry

To update the list of allowed users, you can update the list of allowed users in ./conf/traefik/ldap.yml on apps-test and then apps-prod

Initial bootstrapping on SET managed stage/production VMs)

cd /opt
git clone git@github.com:lehigh-university-libraries/linderman

rollout

A couple files need to be present on the host:

# docker login to auth `docker pull` inside rollout container
/root/.docker/config.json
# deploy token to run `git pull` inside rollout container
/root/.ssh/id_rsa

Setup as a systemd Service

systemd is used to manage the docker compose stack. You can find the unit file in scripts/systemd/linderman.service

cp /opt/linderman/scripts/systemd/linderman.service /etc/systemd/system/
systemctl enable linderman.service

Docker compose liveness probes

Linderman test and prod both use docker-autoheal to ensure the services recover after docker daemon restarts (i.e. OS upgrades) and are healthy. The process is ran using systemd and was installed like so:

$ curl -Lo dah.tar.gz "https://github.com/lehigh-university-libraries/docker-autoheal/releases/download/0.2.8/docker-autoheal_Linux_x86_64.tar.gz"
$ tar -zxvf dah.tar.gz
$ sudo mv docker-autoheal /usr/bin/
$ sudo systemctl enable docker-autoheal.service
$ sudo systemctl start docker-autoheal.service

TLS Certs

Traefik is configured to use Lehigh's wildcard cert. When copying the cert for traefik, ensure the full chain is in ./certs/cert.pem

cd /opt/linderman
cat /etc/ssl/certs/lib.lehigh.edu.crt /etc/ssl/certs/gd_bundle-g2-g1.crt | sudo tee certs/cert.pem