Repository navigation
Commit 1690426
authored
Improve local integration test workflow (#6873)
* Harden database container healthchecks with TCP probes and startup intervals
During first boot initialization, the official MySQL and PostgreSQL
container entrypoints start temporary bootstrap servers listening only
on Unix domain sockets (--skip-networking / listen_addresses='').
Probing with `mysqladmin ping -h localhost` and `pg_isready -U postgres`
uses Unix domain sockets and reports healthy prematurely against the
temporary servers before they are shut down and restarted with TCP networking.
1. Probe over TCP explicitly:
- MySQL: mysqladmin ping -h 127.0.0.1 --protocol=TCP --silent
- PostgreSQL: pg_isready -U postgres -h 127.0.0.1
(MSSQL already probes over TCP via sqlcmd with a SELECT 1 query).
2. Configure startup grace period and cadence:
- start_period: 60s absorbs slow initialization (such as MSSQL creating
system databases) without penalizing fast boots or consuming retries.
- start_interval: 2s polls frequently during bootstrap so container
readiness is detected promptly once TCP accepts connections.
3. Relax steady-state health checks and unify retries:
- interval: 120s minimizes background process forks and connection overhead
during test execution, since the test suite itself actively queries the
databases and detects connection drops immediately.
- retries: 1 marks a container unhealthy immediately if a scheduled probe
fails outside bootstrap, without waiting multiple 120s interval cycles.
- Unify healthcheck parameters identically across PostgreSQL, MySQL, and MSSQL.
* Remove redundant `--include` when `--only` is used
* Validate database tags in integration tests
In CI, integration test shards use ExUnit's `--only database:<service>`.
If a test declares a `:database` tag with a typo (e.g. `:postgres`
instead of `:postgresql`), ExUnit silently skips the test across all
shards: the `none` shard excludes it because a `:database` tag key
exists, and the database shards skip it because the value does not match
the filter.
Unlike tests that omit the tag entirely (which run on the `none` shard
and fail loudly if they attempt to connect to an unstarted database
container), tests with invalid tags are completely omitted from
execution and summary reporting without warning.
Validate all `@tag database: ...`, `@describetag database: ...`, and
`@moduletag database: ...` declarations in `test_helper.exs` before
tests run.
* Revise integration test summary aggregation
The initial prototype for summarizing and aggregating integration test
results was quickly drafted by an AI coding agent to validate the
sharding concept. While it functioned, it was filled with prototype
"slop": unnecessarily defensive guards, redundant fallback chains (like
runtime checks for Jason vs JSON), mixed concerns across parsing and
presentation, and formatting output meant solely for multi-shard CI
runs.
This commit refactors both the ExUnit formatter and the summary
aggregator with a clearer, layered architecture and human-friendly
observability.
Summary Formatter:
- Fail early if missing required environment variables.
- Output deterministic, pretty-printed, human-readable JSON that is
easier to inspect and debug.
- Rather than naively accumulating tests one by one via
`handle_cast({:test_finished, ...})`, rely on ExUnit's lifecycle
guarantees by reading completed tests directly in `:module_finished`.
Aggregator Script:
- Separate raw JSON ingestion, domain transformation, and Markdown
presentation. This eliminates ad-hoc dictionary navigation and
fallback checks scattered across report rendering.
- Use Elixir's native `JSON.decode!/1` only.
- Unified concept from "Shard" to "Job": Abstract terminology across the
pipeline to "Job". This establishes a consistent naming model that
works cleanly whether aggregating multiple matrix jobs in CI or
inspecting a single local run.
- Stop mutating `$GITHUB_STEP_SUMMARY` directly inside the Elixir
script. The script now expects a directory argument, formats Markdown,
and emits it to stdout, making it equally usable locally or in the CI
workflow.
- Visual timeline & metrics: Reconstruct concurrent module execution
into Mermaid Gantt charts and provide aggregated top-slowest tables
across versions without cluttering the summary.
CI Workflow:
- Isolate intermediate summary artifacts in
`${{ runner.temp }}/summaries` to keep checkout trees clean.
- Add `continue-on-error: true` so reporting issues never mask or fail
actual test suite results.
* Improve local/Docker integration test workflow
Previous work overhauled CI integration tests to use explicit workflow
steps rather than the monolithic `integration_test/test.sh` script,
leaving `test.sh` obsolete and the local container testing experience
outdated.
Meanwhile, local test execution lacked parity with CI, relied on both
Docker Compose and `docker run`, and suffered from database container
restart delays.
This change brings the architectural improvements made in CI to local
development by unifying everything under Docker Compose. It establishes
an isolated container environment that minimizes developer machine
exposure while optimizing for fast, iterative local test cycles.
Docker Compose Architecture & Isolation:
- Replace separate `docker run` and compose commands by consolidating
the test runner and databases into `docker-compose.yml`. This keeps
all containers on a shared, managed Docker network without needing
loosely configured host network bindings or exposed ports on the host.
- Using a custom profile, `docker compose up` continues to only start
database services unless the `runner` profile is specifically invoked,
keeping database lifecycle decoupled from test runs.
- Wait until databases are fully initialized before the test container
runs, avoiding race conditions during startup.
- Run tests as a non-root user in an Ubuntu-based image matching the CI
environment.
- Mount named volumes to isolate build artifacts from the host. This
prevents native C compilation artifacts (e.g. `bcrypt_elixir`,
`argon2_elixir`) from conflicting across host and container OS/arch,
while preserving fast incremental compilation across runs.
- Environment & version sync: Define `integration_test/.env` to
parameterize `ELIXIR_IMAGE_TAG` against the latest CI matrix version,
making multi-version testing trivial via environment overrides.
Local Developer Workflow:
- Database services remain running in the background between test
invocations. Developers can re-run tests immediately without paying
database shutdown/boot latency on every run.
- Explicit teardown: Add `./docker.sh down` to easily clean up resources
when finished.
- Automatically calculate compilation concurrency based on available CPU
cores, bringing test execution times down to ~3m30s on modern
hardware.
- Forward all arguments and flags directly to `mix test`, allowing
developers to target specific files or database tags (e.g.
`./docker.sh --include database:postgresql test/...`).
- Remove the obsolete `integration_test/test.sh` script now that local
runs are handled by `docker.sh`.
* Do not publish database ports to host by default
In local development environments, publishing database ports to all
host interfaces (0.0.0.0) by default risks port collisions with locally
running database instances or other projects, and unnecessarily exposes
services on the local network.
1. Remove default host `ports:` bindings from `docker-compose.yml`.
Containerized test runs (`docker.sh`) connect to databases via
internal Docker bridge networking without requiring host port
exposure.
2. Introduce `docker-compose.ports.yml` with ports bound strictly to
loopback (`127.0.0.1`).
3. Update `ci.yml` to pass `-f docker-compose.ports.yml` because CI runs
`mix test` directly on the host runner.
4. Update `README.md` to document using `-f docker-compose.ports.yml`
for host-native testing.
* Skip MSSQL-dependent tests on unsupported architectures
The official Microsoft SQL Server image only supports the `amd64`
architecture.
On `arm64` machines (e.g., Apple Silicon), running the `amd64` image
is either not possible or likely to crash under emulation.
This makes MSSQL optional in the local Docker workflow:
- Assign the `mssql` Compose profile to the `mssql` service and mark it
as not required under the `runner` service dependencies.
- In `docker.sh`, detect host architecture via `uname -m` and only
enable the `mssql` profile on `amd64`.
- In `CodeGeneratorCase`, provide `reason_if_mssql_unsupported/0` to
skip tests tagged `database: :mssql` on unsupported architectures,
ensuring `mix test --include database` skips MSSQL instead of failing
with connection errors while non-DB tests in the same files continue
to run.
- In `entrypoint.sh`, probe hostname resolution before starting the
background `socat` bridge.
CI runs on `amd64` (`ubuntu-24.04`) and starts database containers on
demand by service name. In Docker Compose, explicitly naming a service
bypasses profile filters, ensuring all MSSQL integration tests continue
to run in CI.
* Increase DB max_connections for higher concurrency
Generated Phoenix applications configure their test repository pool size
based on available CPU cores:
pool_size: System.schedulers_online() * 2
On developer workstations with higher core counts (e.g. 16+ cores),
running multiple async integration test modules in parallel causes total
client connections to exceed the default limits of official container
images (PostgreSQL's default of 100 and MySQL's default of 151), leading
to connection exhaustion failures during test runs.
Raising the ceiling to 300 adds negligible shared memory overhead while
providing sufficient headroom under high concurrency.
MSSQL Server defaults to dynamic user connection management (up to
32,767 connections), therefore requires no adjustment.
* Report test summary in local integration tests
Enable test duration and module timeline tracking for local
containerized integration test runs by writing `summary.json` and
`summary.md` to a timestamped output directory
(`tmp/summary_${TIMESTAMP}/`).
* Discard socat error output
Normal usage prints innocuous error messages interleaved with `mix test`
output like:
2026/09/28 19:29:03 socat[7147] E write(6, 0xaaaad4508000, 55): Broken pipe
If the socat bridge would not be working, tests would fail loudly anyway.
* Bind socat bridge only to loopback interface
* Avoid ExUnit CPU oversubscription in integration tests
While generally useful, the default `--max-cases` being two times the
number of schedulers/cores can lead to CPU oversubscription when running
integration tests which spend significant time doing CPU work.
This also changes CI to compute the number of cores dynamically instead
of using hardcoded values based on the current runner capabilities.1 parent 5a9e92f commit 1690426
20 files changed
Lines changed: 949 additions & 747 deletions
File tree
- .github/workflows
- integration_test
- test
- code_generation
- support
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
236 | 236 | | |
237 | 237 | | |
238 | 238 | | |
| 239 | + | |
239 | 240 | | |
240 | 241 | | |
241 | 242 | | |
242 | 243 | | |
243 | 244 | | |
244 | 245 | | |
245 | | - | |
246 | | - | |
247 | | - | |
248 | | - | |
249 | | - | |
250 | | - | |
251 | 246 | | |
252 | 247 | | |
253 | 248 | | |
254 | 249 | | |
255 | 250 | | |
256 | 251 | | |
257 | 252 | | |
| 253 | + | |
| 254 | + | |
| 255 | + | |
| 256 | + | |
| 257 | + | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
| 261 | + | |
| 262 | + | |
| 263 | + | |
| 264 | + | |
258 | 265 | | |
259 | 266 | | |
260 | 267 | | |
261 | 268 | | |
262 | 269 | | |
263 | 270 | | |
264 | 271 | | |
| 272 | + | |
265 | 273 | | |
266 | | - | |
| 274 | + | |
267 | 275 | | |
268 | 276 | | |
269 | 277 | | |
| |||
297 | 305 | | |
298 | 306 | | |
299 | 307 | | |
300 | | - | |
| 308 | + | |
301 | 309 | | |
302 | 310 | | |
303 | 311 | | |
| 312 | + | |
304 | 313 | | |
305 | 314 | | |
306 | 315 | | |
307 | | - | |
| 316 | + | |
308 | 317 | | |
309 | 318 | | |
310 | 319 | | |
| |||
326 | 335 | | |
327 | 336 | | |
328 | 337 | | |
| 338 | + | |
329 | 339 | | |
330 | 340 | | |
331 | 341 | | |
| |||
344 | 354 | | |
345 | 355 | | |
346 | 356 | | |
347 | | - | |
| 357 | + | |
348 | 358 | | |
349 | 359 | | |
350 | | - | |
351 | | - | |
352 | | - | |
| 360 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
16 | 16 | | |
17 | 17 | | |
18 | 18 | | |
| 19 | + | |
19 | 20 | | |
20 | 21 | | |
21 | 22 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
3 | | - | |
| 3 | + | |
4 | 4 | | |
5 | | - | |
| 5 | + | |
6 | 6 | | |
7 | | - | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
8 | 33 | | |
9 | 34 | | |
10 | 35 | | |
11 | | - | |
| 36 | + | |
12 | 37 | | |
13 | 38 | | |
14 | 39 | | |
15 | | - | |
| 40 | + | |
16 | 41 | | |
17 | 42 | | |
18 | 43 | | |
19 | 44 | | |
20 | 45 | | |
21 | 46 | | |
22 | | - | |
23 | | - | |
24 | | - | |
25 | | - | |
26 | | - | |
| 47 | + | |
27 | 48 | | |
28 | 49 | | |
29 | 50 | | |
30 | | - | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
31 | 64 | | |
32 | 65 | | |
33 | 66 | | |
| |||
0 commit comments