Skip to content

Commit 1690426

Browse files
authored
Improve local integration test workflow (#6873)
* Harden database container healthchecks with TCP probes and startup intervals During first boot initialization, the official MySQL and PostgreSQL container entrypoints start temporary bootstrap servers listening only on Unix domain sockets (--skip-networking / listen_addresses=''). Probing with `mysqladmin ping -h localhost` and `pg_isready -U postgres` uses Unix domain sockets and reports healthy prematurely against the temporary servers before they are shut down and restarted with TCP networking. 1. Probe over TCP explicitly: - MySQL: mysqladmin ping -h 127.0.0.1 --protocol=TCP --silent - PostgreSQL: pg_isready -U postgres -h 127.0.0.1 (MSSQL already probes over TCP via sqlcmd with a SELECT 1 query). 2. Configure startup grace period and cadence: - start_period: 60s absorbs slow initialization (such as MSSQL creating system databases) without penalizing fast boots or consuming retries. - start_interval: 2s polls frequently during bootstrap so container readiness is detected promptly once TCP accepts connections. 3. Relax steady-state health checks and unify retries: - interval: 120s minimizes background process forks and connection overhead during test execution, since the test suite itself actively queries the databases and detects connection drops immediately. - retries: 1 marks a container unhealthy immediately if a scheduled probe fails outside bootstrap, without waiting multiple 120s interval cycles. - Unify healthcheck parameters identically across PostgreSQL, MySQL, and MSSQL. * Remove redundant `--include` when `--only` is used * Validate database tags in integration tests In CI, integration test shards use ExUnit's `--only database:<service>`. If a test declares a `:database` tag with a typo (e.g. `:postgres` instead of `:postgresql`), ExUnit silently skips the test across all shards: the `none` shard excludes it because a `:database` tag key exists, and the database shards skip it because the value does not match the filter. Unlike tests that omit the tag entirely (which run on the `none` shard and fail loudly if they attempt to connect to an unstarted database container), tests with invalid tags are completely omitted from execution and summary reporting without warning. Validate all `@tag database: ...`, `@describetag database: ...`, and `@moduletag database: ...` declarations in `test_helper.exs` before tests run. * Revise integration test summary aggregation The initial prototype for summarizing and aggregating integration test results was quickly drafted by an AI coding agent to validate the sharding concept. While it functioned, it was filled with prototype "slop": unnecessarily defensive guards, redundant fallback chains (like runtime checks for Jason vs JSON), mixed concerns across parsing and presentation, and formatting output meant solely for multi-shard CI runs. This commit refactors both the ExUnit formatter and the summary aggregator with a clearer, layered architecture and human-friendly observability. Summary Formatter: - Fail early if missing required environment variables. - Output deterministic, pretty-printed, human-readable JSON that is easier to inspect and debug. - Rather than naively accumulating tests one by one via `handle_cast({:test_finished, ...})`, rely on ExUnit's lifecycle guarantees by reading completed tests directly in `:module_finished`. Aggregator Script: - Separate raw JSON ingestion, domain transformation, and Markdown presentation. This eliminates ad-hoc dictionary navigation and fallback checks scattered across report rendering. - Use Elixir's native `JSON.decode!/1` only. - Unified concept from "Shard" to "Job": Abstract terminology across the pipeline to "Job". This establishes a consistent naming model that works cleanly whether aggregating multiple matrix jobs in CI or inspecting a single local run. - Stop mutating `$GITHUB_STEP_SUMMARY` directly inside the Elixir script. The script now expects a directory argument, formats Markdown, and emits it to stdout, making it equally usable locally or in the CI workflow. - Visual timeline & metrics: Reconstruct concurrent module execution into Mermaid Gantt charts and provide aggregated top-slowest tables across versions without cluttering the summary. CI Workflow: - Isolate intermediate summary artifacts in `${{ runner.temp }}/summaries` to keep checkout trees clean. - Add `continue-on-error: true` so reporting issues never mask or fail actual test suite results. * Improve local/Docker integration test workflow Previous work overhauled CI integration tests to use explicit workflow steps rather than the monolithic `integration_test/test.sh` script, leaving `test.sh` obsolete and the local container testing experience outdated. Meanwhile, local test execution lacked parity with CI, relied on both Docker Compose and `docker run`, and suffered from database container restart delays. This change brings the architectural improvements made in CI to local development by unifying everything under Docker Compose. It establishes an isolated container environment that minimizes developer machine exposure while optimizing for fast, iterative local test cycles. Docker Compose Architecture & Isolation: - Replace separate `docker run` and compose commands by consolidating the test runner and databases into `docker-compose.yml`. This keeps all containers on a shared, managed Docker network without needing loosely configured host network bindings or exposed ports on the host. - Using a custom profile, `docker compose up` continues to only start database services unless the `runner` profile is specifically invoked, keeping database lifecycle decoupled from test runs. - Wait until databases are fully initialized before the test container runs, avoiding race conditions during startup. - Run tests as a non-root user in an Ubuntu-based image matching the CI environment. - Mount named volumes to isolate build artifacts from the host. This prevents native C compilation artifacts (e.g. `bcrypt_elixir`, `argon2_elixir`) from conflicting across host and container OS/arch, while preserving fast incremental compilation across runs. - Environment & version sync: Define `integration_test/.env` to parameterize `ELIXIR_IMAGE_TAG` against the latest CI matrix version, making multi-version testing trivial via environment overrides. Local Developer Workflow: - Database services remain running in the background between test invocations. Developers can re-run tests immediately without paying database shutdown/boot latency on every run. - Explicit teardown: Add `./docker.sh down` to easily clean up resources when finished. - Automatically calculate compilation concurrency based on available CPU cores, bringing test execution times down to ~3m30s on modern hardware. - Forward all arguments and flags directly to `mix test`, allowing developers to target specific files or database tags (e.g. `./docker.sh --include database:postgresql test/...`). - Remove the obsolete `integration_test/test.sh` script now that local runs are handled by `docker.sh`. * Do not publish database ports to host by default In local development environments, publishing database ports to all host interfaces (0.0.0.0) by default risks port collisions with locally running database instances or other projects, and unnecessarily exposes services on the local network. 1. Remove default host `ports:` bindings from `docker-compose.yml`. Containerized test runs (`docker.sh`) connect to databases via internal Docker bridge networking without requiring host port exposure. 2. Introduce `docker-compose.ports.yml` with ports bound strictly to loopback (`127.0.0.1`). 3. Update `ci.yml` to pass `-f docker-compose.ports.yml` because CI runs `mix test` directly on the host runner. 4. Update `README.md` to document using `-f docker-compose.ports.yml` for host-native testing. * Skip MSSQL-dependent tests on unsupported architectures The official Microsoft SQL Server image only supports the `amd64` architecture. On `arm64` machines (e.g., Apple Silicon), running the `amd64` image is either not possible or likely to crash under emulation. This makes MSSQL optional in the local Docker workflow: - Assign the `mssql` Compose profile to the `mssql` service and mark it as not required under the `runner` service dependencies. - In `docker.sh`, detect host architecture via `uname -m` and only enable the `mssql` profile on `amd64`. - In `CodeGeneratorCase`, provide `reason_if_mssql_unsupported/0` to skip tests tagged `database: :mssql` on unsupported architectures, ensuring `mix test --include database` skips MSSQL instead of failing with connection errors while non-DB tests in the same files continue to run. - In `entrypoint.sh`, probe hostname resolution before starting the background `socat` bridge. CI runs on `amd64` (`ubuntu-24.04`) and starts database containers on demand by service name. In Docker Compose, explicitly naming a service bypasses profile filters, ensuring all MSSQL integration tests continue to run in CI. * Increase DB max_connections for higher concurrency Generated Phoenix applications configure their test repository pool size based on available CPU cores: pool_size: System.schedulers_online() * 2 On developer workstations with higher core counts (e.g. 16+ cores), running multiple async integration test modules in parallel causes total client connections to exceed the default limits of official container images (PostgreSQL's default of 100 and MySQL's default of 151), leading to connection exhaustion failures during test runs. Raising the ceiling to 300 adds negligible shared memory overhead while providing sufficient headroom under high concurrency. MSSQL Server defaults to dynamic user connection management (up to 32,767 connections), therefore requires no adjustment. * Report test summary in local integration tests Enable test duration and module timeline tracking for local containerized integration test runs by writing `summary.json` and `summary.md` to a timestamped output directory (`tmp/summary_${TIMESTAMP}/`). * Discard socat error output Normal usage prints innocuous error messages interleaved with `mix test` output like: 2026/09/28 19:29:03 socat[7147] E write(6, 0xaaaad4508000, 55): Broken pipe If the socat bridge would not be working, tests would fail loudly anyway. * Bind socat bridge only to loopback interface * Avoid ExUnit CPU oversubscription in integration tests While generally useful, the default `--max-cases` being two times the number of schedulers/cores can lead to CPU oversubscription when running integration tests which spend significant time doing CPU work. This also changes CI to compute the number of cores dynamically instead of using hardcoded values based on the current runner capabilities.
1 parent 5a9e92f commit 1690426

20 files changed

Lines changed: 949 additions & 747 deletions

‎.github/workflows/ci.yml‎

Lines changed: 21 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -236,34 +236,42 @@ jobs:
236236
- elixir: "1.18.4"
237237
otp: "27.3.4.3"
238238

239+
# keep version in sync with integration_test/.env
239240
- elixir: "1.20.4"
240241
otp: "29.0.5"
241242

242243
env:
243244
ELIXIR_ASSERT_TIMEOUT: 10000
244245
PHX_CI: true
245-
# Set to half of the runner vCPU cores to compile deps/NIFs
246-
# concurrently across OS processes without CPU oversubscription.
247-
# See https://docs.github.com/en/actions/reference/runners/github-hosted-runners#standard-github-hosted-runners-for-public-repositories
248-
# and https://elixir-lang.org/blog/2025/10/16/elixir-v1-19-0-released/
249-
MIX_OS_DEPS_COMPILE_PARTITION_COUNT: 2
250-
MAKEFLAGS: "-j2"
251246

252247
steps:
253248
- name: Checkout
254249
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
255250
with:
256251
persist-credentials: false
257252

253+
- name: Configure runner concurrency
254+
run: |
255+
CORES=$(nproc)
256+
HALF_CORES=$(( CORES > 2 ? CORES / 2 : 2 ))
257+
echo "CORES=$CORES" >> "$GITHUB_ENV"
258+
echo "HALF_CORES=$HALF_CORES" >> "$GITHUB_ENV"
259+
# Set to half of the runner vCPU cores to compile deps/NIFs
260+
# concurrently across OS processes without CPU oversubscription.
261+
# See https://elixir-lang.org/blog/2025/10/16/elixir-v1-19-0-released/
262+
echo "MIX_OS_DEPS_COMPILE_PARTITION_COUNT=$HALF_CORES" >> "$GITHUB_ENV"
263+
echo "MAKEFLAGS=-j$HALF_CORES" >> "$GITHUB_ENV"
264+
258265
- name: Mount installer/tmp on tmpfs
259266
run: |
260267
mkdir -p installer/tmp
261268
sudo mount -t tmpfs -o size=4G,uid=$(id -u),gid=$(id -g) tmpfs installer/tmp
262269
263270
- name: Start database container (${{ matrix.service }})
264271
if: matrix.service != 'none'
272+
working-directory: integration_test
265273
run: |
266-
docker compose -f integration_test/docker-compose.yml up -d --wait ${{ matrix.service }}
274+
docker compose -f docker-compose.yml -f docker-compose.ports.yml up -d --wait ${{ matrix.service }}
267275
268276
- name: Set up Elixir
269277
uses: erlef/setup-beam@54075bcc5e249e4758d363f27d099f55d843f124 # v1.24.1
@@ -297,14 +305,15 @@ jobs:
297305
if [ "${{ matrix.service }}" = "none" ]; then
298306
FLAGS="--include database:sqlite3"
299307
else
300-
FLAGS="--include database:${{ matrix.service }} --only database:${{ matrix.service }}"
308+
FLAGS="--only database:${{ matrix.service }}"
301309
fi
302310
303311
mix test $FLAGS \
312+
--max-cases "$CORES" \
304313
--formatter ExUnit.CLIFormatter \
305314
--formatter Phoenix.Integration.SummaryFormatter
306315
env:
307-
PHX_TEST_SERVICE: ${{ matrix.service }}
316+
PHX_TEST_JOB: ${{ matrix.service }}
308317
PHX_ELIXIR_VERSION: ${{ matrix.versions.elixir }}
309318
PHX_OTP_VERSION: ${{ matrix.versions.otp }}
310319
PHX_INTEGRATION_SUMMARY_JSON: ${{ runner.temp }}/summary-${{ matrix.service }}-${{ matrix.versions.otp }}-${{ matrix.versions.elixir }}.json
@@ -326,6 +335,7 @@ jobs:
326335
timeout-minutes: 2
327336
needs: integration-test-elixir
328337
if: always()
338+
continue-on-error: true
329339

330340
steps:
331341
- name: Checkout
@@ -344,9 +354,7 @@ jobs:
344354
with:
345355
pattern: summary-*
346356
merge-multiple: true
347-
path: tmp/summaries
357+
path: ${{ runner.temp }}/summaries
348358

349359
- name: Aggregate test summaries
350-
run: elixir integration_test/aggregate_summary.exs tmp/summaries
351-
env:
352-
MATRIX_RESULT: ${{ needs.integration-test-elixir.result }}
360+
run: elixir integration_test/aggregate_summary.exs "${{ runner.temp }}/summaries" >> "$GITHUB_STEP_SUMMARY"

‎.gitignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@
1616

1717
/integration_test/_build/
1818
/integration_test/deps/
19+
/integration_test/tmp/
1920

2021
erl_crash.dump
2122
phoenix-*.ez

‎integration_test/.env‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
# Default image tag for integration test runner (must be an Ubuntu-based hexpm/elixir image)
2+
# Keep Elixir and Erlang/OTP versions in sync with the latest matrix entry in .github/workflows/ci.yml
3+
#
4+
# Find Ubuntu image tags on Bob:
5+
# - Web UI: https://bob.hex.pm/docker?repo=hexpm/elixir&os=ubuntu&os_version=noble-&sort=elixir_version,erlang_version,os_version
6+
# - API: https://bob.hex.pm/api/docker?repo=hexpm/elixir&os=ubuntu&os_version=noble-&sort=elixir_version,erlang_version,os_version
7+
ELIXIR_IMAGE_TAG=1.20.4-erlang-29.0.5-ubuntu-noble-20260810

‎integration_test/Dockerfile‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
ARG ELIXIR_IMAGE_TAG
2+
3+
FROM hexpm/elixir:${ELIXIR_IMAGE_TAG}
4+
5+
RUN apt-get update \
6+
&& apt-get install -y --no-install-recommends build-essential git socat \
7+
&& rm -rf /var/lib/apt/lists/* \
8+
&& if id -u ubuntu >/dev/null 2>&1; then \
9+
usermod -l runner -d /home/runner -m ubuntu && groupmod -n runner ubuntu; \
10+
else \
11+
useradd -m -u 1000 -s /bin/bash runner; \
12+
fi \
13+
&& mkdir -p /home/runner/.mix /home/runner/.hex /phoenix/integration_test/_build /phoenix/integration_test/deps /phoenix/installer/tmp \
14+
&& chown -R runner:runner /home/runner /phoenix
15+
16+
USER runner
17+
WORKDIR /phoenix/integration_test
18+
19+
RUN mix local.hex --force \
20+
&& mix local.rebar --force
21+
22+
COPY --chmod=755 entrypoint.sh /entrypoint.sh
23+
24+
ENTRYPOINT ["/entrypoint.sh"]
25+
CMD ["--include", "database"]

‎integration_test/README.md‎

Lines changed: 44 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -1,33 +1,66 @@
11
## Phoenix Integration Tests
22

3-
This project contains integration tests for phoenix's generated projects.
3+
This project contains integration tests for Phoenix's generated projects.
44

5-
## Running tests
5+
## Running Tests
66

7-
To install dependencies, run:
7+
### Option A: Using Docker & Docker Compose (Recommended)
8+
9+
Run the integration test suite in an isolated container without needing local database or Elixir setup:
10+
11+
$ ./docker.sh
12+
13+
Pass specific test files or tag filters to target individual tests:
14+
15+
$ ./docker.sh test/code_generation/app_with_defaults_test.exs --include database
16+
$ ./docker.sh --include database:postgresql
17+
18+
To test against different Elixir or Erlang/OTP versions:
19+
20+
$ ELIXIR_IMAGE_TAG=1.18.5-erlang-27.3.4.17-ubuntu-noble-20260810 ./docker.sh
21+
22+
Find valid image tags (Ubuntu-based, `hexpm/elixir` repo) on [bob.hex.pm](https://bob.hex.pm/docker?repo=hexpm/elixir&os=ubuntu&os_version=noble-&sort=elixir_version,erlang_version,os_version).
23+
24+
All databases (PostgreSQL, MySQL, and MSSQL) are started by default. Because the official MSSQL image supports only `amd64`, the MSSQL service and tests are automatically skipped on other platforms (e.g., `arm64`).
25+
26+
When running tests repeatedly, backing database containers remain warm in the background for instant re-runs. To tear down the database services when finished:
27+
28+
$ ./docker.sh down
29+
30+
### Option B: Running Natively on the Host
31+
32+
Install dependencies and run the tests on your machine:
833

934
$ mix deps.get
1035

11-
Then run the basic test suite (no dependencies on the database) with:
36+
Run the basic test suite (without databases):
1237

1338
$ mix test
1439

15-
To run the test suite with tests that test a specific database, run:
40+
To run tests against a specific database:
1641

1742
$ mix test --include database:postgresql
1843
$ mix test --include database:mysql
1944
$ mix test --include database:mssql
2045
$ mix test --include database:sqlite3
2146

22-
For convenience, there is also a `docker-compose.yml` file that allows for starting up all of the supported databases locally.
23-
24-
$ docker-compose up
25-
26-
This allows all tests to be run with the following command
47+
To run all tests including databases:
2748

2849
$ mix test --include database
2950

30-
Or alternatively, with docker and docker compose installed, you can just run `./docker.sh`.
51+
> [!NOTE]
52+
> **Running databases for host-native tests:**
53+
> `docker-compose.yml` does not publish database ports to the host by default. This avoids port collisions with local databases already running on your machine (e.g. PostgreSQL on 5432 or MySQL on 3306) and prevents exposing test databases with default credentials to the local network.
54+
>
55+
> When running tests directly on the host (`mix test`) rather than inside Docker (`./docker.sh`), include `docker-compose.ports.yml` to publish ports strictly to loopback (`127.0.0.1`):
56+
>
57+
> ```bash
58+
> # Start all databases with host ports published on 127.0.0.1
59+
> $ docker compose -f docker-compose.yml -f docker-compose.ports.yml up -d
60+
>
61+
> # Or start only a specific database (e.g. Postgres)
62+
> $ docker compose -f docker-compose.yml -f docker-compose.ports.yml up -d postgres
63+
> ```
3164
3265
## How tests are written
3366

0 commit comments

Comments
 (0)