Skip to content

Commit 777ef53

Browse files
krab1kkrab1k
committed
docs: document Ubuntu deployment setup and maintenance
1 parent bc4f774 commit 777ef53

3 files changed

Lines changed: 256 additions & 78 deletions

File tree

‎docs/deployment/README.md‎

Lines changed: 169 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -1,20 +1,175 @@
11
# Deployment
22

3-
- [Automatic Certificate Renewal](./certificate-renewal.md)
3+
The [Deploy (Manual) workflow](../../.github/workflows/deploy-manual.yml) updates an
4+
existing installation over SSH. It does **not** provision a new server or start
5+
PostgreSQL. Follow the bootstrap instructions before using it on a fresh host.
46

5-
## Deploying to production
6-
`Deploy to Production` GitHub Action provides a semi-automatic way to deploy new ACC III version, implemented [here](../../.github/workflows/deploy.yml). It connects to the server via SSH and rebuilds the configured containers using `docker compose`. It is triggered on `workflow_dispatch` (i.e., manual trigger).
7+
## Prepare an Ubuntu LTS host
78

8-
You can trigger the deploy workflow by going to *Actions -> Deploy to Production* and choosing which branch and configuration to use during the deployment:
9-
![deploy trigger](./images/deploy-trigger.png)
9+
These instructions assume an x86-64 Ubuntu LTS server with systemd, the SSH user
10+
`ubuntu`, and one ACC installation per host. Paths and container names are fixed
11+
in Compose and the workflow; selecting Development or Production does not isolate
12+
two installations on one host. ChargeFW2 currently contains x86-64-specific paths.
1013

11-
### Configurations
12-
- `all` - rebuild everything
13-
- `api` - rebuild only backend (FastAPI)
14-
- `api including chargefw2` - rebuild API container and ChargeFW2
15-
- `web` - rebuild only frontend
16-
- `nginx` - rebuild only nginx container
14+
1. Install Git, OpenSSH server, `ca-certificates`, `curl`, `util-linux` (for `flock`),
15+
`logrotate`, and `certbot` using Ubuntu packages. Install Docker Engine and the
16+
Compose v2/Buildx plugins using the [official Ubuntu instructions](https://docs.docker.com/engine/install/ubuntu/).
17+
2. Enable Docker, SSH, and `logrotate.timer`. Grant `ubuntu` Docker access and
18+
reconnect before testing `docker ps` and `docker compose version`. Docker access
19+
is effectively root access. The workflow also requires noninteractive `sudo`
20+
for its Git commands; verify this setup before deployment.
21+
3. Configure SSH key access for GitHub Actions. Ensure the server can fetch the
22+
repository and submodule, including when Git is invoked through `sudo`.
23+
4. Point the configured DNS names at the host. Permit SSH and TCP 80/443; do not
24+
run a host web server that occupies those ports.
1725

18-
### Secrets
19-
Secrets are set in *Settings -> Secrets and variables -> Actions*:
20-
![secrets](./images/secrets.png)
26+
**Network warning:** the current deployment Compose file also publishes database
27+
port 5432, API port 8000, and frontend port 3000 on all interfaces. Restrict these
28+
through the infrastructure firewall or appropriately reviewed Docker-aware rules.
29+
Do not assume UFW alone blocks Docker-published ports.
30+
31+
Clone the repository to `/home/ubuntu/AtomicChargeCalculator`, then run from that
32+
checkout (replace the clone URL with the repository URL if creating it):
33+
34+
```bash
35+
git submodule update --init --recursive
36+
sudo install -d -m 0755 /home/acc /var/log/acc /var/www/certbot
37+
sudo install -d -m 0755 -o root -g adm /var/log/acc/nginx
38+
sudo touch /var/log/acc/nginx/access.log /var/log/acc/nginx/error.log
39+
sudo chgrp adm /var/log/acc/nginx/access.log /var/log/acc/nginx/error.log
40+
sudo chmod 0640 /var/log/acc/nginx/access.log /var/log/acc/nginx/error.log
41+
sudo install -m 0644 -o root -g root src/deployment/nginx/logrotate /etc/logrotate.d/acc-nginx
42+
sudo systemctl enable --now logrotate.timer
43+
sudo logrotate --debug /etc/logrotate.d/acc-nginx
44+
```
45+
46+
Inspect existing directories before changing permissions on an established host.
47+
The logrotate debug command does not change files or test the reopen hook.
48+
49+
## Environment and secrets
50+
51+
Configure GitHub environments named **Development** and **Production**, with the
52+
appropriate deployment secrets:
53+
54+
| Secret | Purpose |
55+
| --- | --- |
56+
| `SSH_HOST` | Target server |
57+
| `SSH_USERNAME` | Deployment user, normally `ubuntu` |
58+
| `SSH_PRIVATE_KEY` | Key authorized on that server |
59+
| `SSH_PORT` | Optional; defaults to 22 |
60+
| `DB_PASSWORD` | PostgreSQL password |
61+
| `OIDC_CLIENT_ID` | Registered Life Science client |
62+
| `OIDC_CLIENT_SECRET` | Life Science client secret |
63+
64+
For manual bootstrap, securely export `DB_PASSWORD`, `OIDC_CLIENT_ID`, and
65+
`OIDC_CLIENT_SECRET` in the shell as well. Do not commit credentials or include
66+
them in shared terminal output. The workflow does not create a server `.env` file.
67+
Current workflow exports interpolate secrets into shell commands, and the database
68+
password is embedded in a connection URL. Shell-special characters and URL-reserved
69+
characters are not safely supported by that implementation; review compatibility
70+
before deployment rather than weakening credential strength.
71+
72+
For **Development**, also export:
73+
74+
```bash
75+
export VITE_BASE_API_URL=https://acc-dev.biodata.ceitec.cz:443/api/v1
76+
export OIDC_BASE_URL=https://acc-dev.biodata.ceitec.cz/
77+
export OIDC_REDIRECT_URL=https://acc-dev.biodata.ceitec.cz/api/v1/auth/callback
78+
export NGINX_CONFIG_FILE=nginx.dev.conf
79+
```
80+
81+
For **Production**, use:
82+
83+
```bash
84+
export VITE_BASE_API_URL=https://acc.biodata.ceitec.cz:443/api/v1
85+
export OIDC_BASE_URL=https://acc.biodata.ceitec.cz/
86+
export OIDC_REDIRECT_URL=https://acc.biodata.ceitec.cz/api/v1/auth/callback
87+
export NGINX_CONFIG_FILE=nginx.prod.conf
88+
```
89+
90+
Register the matching callback with Life Science. The frontend API URL is baked
91+
into its image at build time. Both deployments use the database name `postgres_prod`
92+
and user `prod_user` on their respective hosts. Changing `DB_PASSWORD` does not
93+
change the password in an already initialized PostgreSQL volume.
94+
95+
## First deployment
96+
97+
Obtain [initial TLS certificates](./certificate-renewal.md) before starting Nginx.
98+
The production configuration needs a certificate covering both public hostnames.
99+
100+
From the repository root, with the environment above set:
101+
102+
```bash
103+
docker build -t chargefw2-base:acc -f ChargeFW2/Dockerfile ChargeFW2
104+
cd src/deployment
105+
docker compose -f docker-compose.yml -f docker-compose.prod.yml pull nginx
106+
docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d db
107+
docker compose -f docker-compose.yml -f docker-compose.prod.yml ps db
108+
```
109+
110+
Wait until PostgreSQL is healthy, then build and start the application:
111+
112+
```bash
113+
docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d --build api web
114+
docker logs --tail 100 acc-api
115+
docker compose -f docker-compose.yml -f docker-compose.prod.yml run --rm --no-deps --pull never nginx nginx -t
116+
docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d --no-deps --pull never nginx
117+
```
118+
119+
Check API startup logs for successful database migrations and application startup;
120+
a running container alone does not establish that migrations succeeded. Verify
121+
HTTPS, API calls, example viewing, and login. Complete certificate renewal setup.
122+
123+
Use **both** Compose files on hosted dev and production. Bare `docker compose`
124+
loads the local-development override. Nginx preflight needs existing certificates,
125+
host log setup, a locally available Nginx image, and running/resolvable `api` and
126+
`web` services. Consequently, selecting every GitHub deployment option on an empty
127+
host is not a substitute for this bootstrap.
128+
129+
## Subsequent GitHub deployments
130+
131+
Open **Actions > Deploy (Manual) > Run workflow**. Select the target environment,
132+
the workflow revision, the branch to deploy, and the required checkboxes:
133+
134+
- **Rebuild ChargeFW2:** builds the base image; also select **Deploy API** to use it.
135+
- **Deploy API:** builds and recreates the API container.
136+
- **Deploy Web:** builds and recreates the frontend container.
137+
- **Deploy Nginx:** checks logging prerequisites and tests the candidate Nginx
138+
configuration before service updates, then recreates Nginx.
139+
140+
All checkboxes default to false. The [deployment logic](../../.github/workflows/deploy-logic.yml)
141+
resets the server checkout to the selected branch; do not keep uncommitted work
142+
there. A host lock rejects overlapping workflow deployments. Deployments are not
143+
zero-downtime and API recreation can interrupt jobs. Use a maintenance window when
144+
necessary. Nginx uses `--pull never`; explicitly pull a chosen image before deploying
145+
an image update. Restarting a container does not apply changed Compose mounts.
146+
147+
## Logging maintenance
148+
149+
After deployment, verify that website requests appear as JSON in
150+
`/var/log/acc/nginx/access.log`. Check that `logrotate.timer` remains active and
151+
monitor available disk space. To test rotation, use the development server:
152+
153+
```bash
154+
sudo logrotate --force /etc/logrotate.d/acc-nginx
155+
```
156+
157+
Make another request and confirm it appears in the replacement `access.log`.
158+
This exercises the log reopen hook without restarting Nginx. Normal rotation is
159+
scheduled automatically; forced rotation is only a verification step.
160+
161+
## Persistence and retention
162+
163+
- Nginx access/error logs live in host `/var/log/acc/nginx`; monthly rotation keeps
164+
24 archives and compresses older files. Empty files are skipped. This is not a
165+
hard size limit; monitor disk space and copy historical logs off-server.
166+
- Nginx stdout/stderr remains available through `docker logs`, bounded by Docker's
167+
configured rotation. These diagnostics are removed with the container;
168+
persistent access/error files are separate from Docker logs.
169+
- API logs remain in host `/var/log/acc/logs.log`; the Nginx rotation rule does not
170+
rotate that file. Calculation data remains under host `/home/acc`.
171+
- PostgreSQL uses named volume `deployment_db-data` with the default project name.
172+
Never use `docker compose down -v` or remove that volume to perform an update.
173+
174+
Host logs and volumes survive container recreation, not host loss. Arrange and test
175+
off-server backups separately.

‎docs/deployment/certificate-renewal.md‎

Lines changed: 59 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -1,38 +1,76 @@
1-
# Certificate Renewal
2-
The following doc describes a way to automatically renew SSL certificates using `certbot` and `systemd`.
1+
# TLS Certificates
32

4-
## Systemd Configuration
5-
Certbot relies on `certbot.timer` and `certbot.service` services to automatically renew certificates. Verify that both are running on your system:
3+
Certbot runs on the Ubuntu host. Nginx runs in Docker and mounts `/etc/letsencrypt`
4+
and `/var/www/certbot`. Do not use `certbot --nginx` to configure this container.
5+
6+
## Initial issuance
7+
8+
Set DNS and allow inbound TCP 80 before issuance. On a fresh host, with port 80
9+
free, use standalone validation. For development:
610

711
```bash
8-
$ sudo systemctl status certbot.timer
9-
$ sudo systemctl status certbot.service
12+
sudo certbot certonly --standalone --cert-name acc-dev.biodata.ceitec.cz -d acc-dev.biodata.ceitec.cz
13+
```
1014

11-
# List running certbot timers:
12-
$ sudo systemctl list-timers | grep certbot
15+
For production:
16+
17+
```bash
18+
sudo certbot certonly --standalone --cert-name acc2.ncbr.muni.cz -d acc2.ncbr.muni.cz -d acc.biodata.ceitec.cz
1319
```
1420

15-
### Webroot Configuration
16-
So that we don't have to stop the nginx container to free up the port 80 (used by certbot for certificate renewal), we can configure it to use `webroot` authenticator instead.
21+
Production Nginx uses the `acc2.ncbr.muni.cz` certificate directory for both names;
22+
the certificate must cover both. Supply the requested contact information and agree
23+
to the CA terms. Do not rerun standalone issuance against an existing busy server
24+
without arranging port availability. Existing valid certificates need no reissue.
1725

18-
Use the configuration from [domain.conf](/deployment/certificates/renewal/domain.conf) in `/etc/letsencrypt/renewal/<your-domain>.conf` on your system.
26+
Once certificates exist, finish the [first deployment](./README.md#first-deployment).
1927

20-
This will use the already running nginx for verification, instead of spinning up a temporary HTTP server in the case of `standalone` authenticator.
28+
## Webroot renewal
29+
30+
After Nginx is serving HTTP, switch each certificate to webroot renewal so future
31+
renewals do not stop the website. With a Certbot version supporting `reconfigure`:
2132

22-
## Reloading Nginx
23-
Once the certificate is renewed (deployed) we need to reload Nginx configuration. This can be achieved with the `deploy` certbot hook. Add the [reload-nginx.sh](/deployment/certificates/renewal-hooks/deploy/reload-nginx.sh) script to the `renewal-hooks` directory on your system:
2433
```bash
25-
$ cp deployment/certificates/renewal-hooks/deploy/reload-nginx.sh /etc/letsencrypt/renewal-hooks/deploy
26-
$ chmod +x /etc/letsencrypt/renewal-hooks/deploy
34+
sudo certbot reconfigure --cert-name acc-dev.biodata.ceitec.cz --authenticator webroot --webroot-path /var/www/certbot
2735
```
2836

29-
> _Note:_ Don't forget to adjust the path to the `docker-compose.yml` file on your system, if it differs.
37+
Use `acc2.ncbr.muni.cz` instead on production. Check `certbot reconfigure --help`;
38+
if the Ubuntu package lacks this command, follow the installed version's documented
39+
renewal-parameter procedure or upgrade Certbot. The repository
40+
[domain.conf](../../src/deployment/certificates/renewal/domain.conf) shows renewal
41+
parameters only: **do not overwrite the entire generated renewal file with it**.
42+
43+
Nginx serves `/.well-known/acme-challenge/` from `/var/www/certbot` on HTTP port 80.
44+
DNS and that port must remain reachable for renewal.
45+
46+
## Reload hook and scheduler
47+
48+
From the repository root:
49+
50+
```bash
51+
sudo install -d /etc/letsencrypt/renewal-hooks/deploy
52+
sudo install -m 0755 src/deployment/certificates/renewal-hooks/deploy/reload-nginx.sh /etc/letsencrypt/renewal-hooks/deploy/reload-nginx.sh
53+
sudo systemctl enable --now certbot.timer
54+
systemctl status certbot.timer
55+
```
56+
57+
The hook discovers a running container whose name contains `nginx` and reloads its
58+
configuration. It assumes one such container on the host. No Compose path needs
59+
editing. `certbot.service` is a oneshot; it need not stay active between timer runs.
60+
61+
## Verification
62+
63+
```bash
64+
sudo certbot renew --dry-run
65+
```
3066

31-
## Testing
32-
To test if the setup works, you can force the certificate renewal:
67+
A dry run checks renewal but does not normally execute deploy hooks. Separately,
68+
after validating the running Nginx configuration, test the installed reload hook:
3369

3470
```bash
35-
$ sudo certbot renew --cert-name <your-domain> --force-renew
71+
docker exec acc-nginx nginx -t
72+
sudo /etc/letsencrypt/renewal-hooks/deploy/reload-nginx.sh
3673
```
3774

38-
If everything worked as expected, you should see the new certificate (e.g. when navigating to the page).
75+
Inspect command output for successful renewal and reload. Do not routinely force
76+
real certificate renewals just to test setup; that can consume CA rate limits.

‎src/deployment/README.md‎

Lines changed: 28 additions & 43 deletions
Original file line numberDiff line numberDiff line change
@@ -1,59 +1,44 @@
11
# Deployment
2-
How to use the *Deploy to Production* GitHub Action to deploy a new ACC III version is described [here](../../docs/deployment/README.md).
32

4-
## Nginx
3+
For hosted development or production, follow the
4+
[Ubuntu host setup, bootstrap, and GitHub Actions guide](../../docs/deployment/README.md).
5+
The deployment workflow updates an existing installation; it does not provision it.
56

6-
`nginx.conf` provides a configuration that can be used to deploy ACC III using nginx. Reusable declarations are located in `./snippets` to avoid unnecessary duplicity.
7+
## Local Docker setup
78

8-
## SSL Configuration
9-
A simple way is to use certbot (Let's Encrypt):
9+
Run from the repository root:
1010

11-
```bash
12-
$ certbot --nginx -d <url>
13-
```
14-
15-
## Setup Using Docker
16-
A quicker way of running ACC III locally is to use `docker compose`. This will start up 3 containers for API, web and database.
17-
18-
*Note:* If you are using Windows, you might encounter the following error during `api` container startup:
19-
```
20-
exec /acc/entrypoint.sh: no such file or directory
21-
```
22-
23-
This is because of line endings and can be fixed by running `dos2unix` on the `src/backend/entrypoint.sh` file.
24-
25-
### Prepare base ChargeFW2 image
26-
We first need to build the base ChargeFW2 image which is used by the api container:
2711
```bash
28-
$ cd ../../
29-
$ docker build -t chargefw2-base:local -f ./src/backend/Dockerfile.chargefw2 .
12+
git submodule update --init --recursive
13+
docker build -t chargefw2-base:acc -f ChargeFW2/Dockerfile ChargeFW2
14+
cd src/deployment
15+
docker compose up -d --build
3016
```
3117

32-
### Start
33-
Use `-d` if you want the containers to start in the background.
18+
This starts four services: PostgreSQL, API, frontend, and Nginx. Access the UI through
19+
`http://localhost:8080`. Local OIDC credentials and callback settings are placeholders;
20+
working Life Science login needs a registered client and appropriate configuration.
3421

35-
```bash
36-
$ cd ./src/deployment
37-
$ docker compose up -d
38-
```
22+
To stop the local stack without deleting database volumes:
3923

40-
### Stop
4124
```bash
42-
$ docker compose down
25+
docker compose down
4326
```
4427

45-
### Production startup
46-
In order to run compose in production mode, you need to specify which compose files should be used:
28+
On Windows, an API startup error such as `exec /acc/entrypoint.sh: no such file or
29+
directory` can result from CRLF line endings. Ensure `src/backend/entrypoint.sh`
30+
uses Unix line endings.
4731

48-
```bash
49-
$ docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d
50-
```
32+
## Configuration files
5133

52-
*Notes*:
53-
- This will not work locally as the frontend app will try to make requests to the production server.
54-
- Environment variable `DB_PASSWORD` needs to be specified (any password).
34+
- `docker-compose.yml`: base services and PostgreSQL volume.
35+
- `docker-compose.override.yml`: automatically loaded local-development settings.
36+
- `docker-compose.prod.yml`: explicitly selected for both hosted dev and production;
37+
requires host directories, credentials, and certificates described in the guide.
38+
- `nginx/nginx.local.conf`, `nginx/nginx.dev.conf`, `nginx/nginx.prod.conf`: Nginx
39+
configurations with shared declarations under `nginx/snippets/`.
40+
- `nginx/logrotate`: host rotation rule, installed as `/etc/logrotate.d/acc-nginx`.
5541

56-
### Compose Files
57-
- `docker-compose.yml` base configuration.
58-
- `docker-compose.override.yml` development configuration. Is used automatically.
59-
- `docker-compose.prod.yml` production configuration. Needs to be explicitly specified (as described above)
42+
For hosted deployments always specify `-f docker-compose.yml -f docker-compose.prod.yml`.
43+
Do not use the host Certbot Nginx plugin to configure the container; follow the
44+
[initial certificate and renewal instructions](../../docs/deployment/certificate-renewal.md).

0 commit comments

Comments
 (0)