WIP: ephemeral environment for PRs - #22
Closed
elookpotts-nvidia wants to merge 1 commit into
Closed
Conversation
elookpotts-nvidia
temporarily deployed
to
ephemeral-pr-22
November 18, 2025 01:58 — with 
GitHub Actions
Inactive
GitHub Actions
Inactive
elookpotts-nvidia
force-pushed
the
elookpotts/ephemeral-env
branch
from
November 18, 2025 02:01
49928a1 to
a0349a5
Compare
elookpotts-nvidia
changed the title
3 tasks
jiaenren
added a commit
that referenced
this pull request
Jun 16, 2026
* ci: add Local KIND Deployment workflow for OETF KIND e2e validation Runs `bazel run //test/oetf:deploy_and_run -- --env kind --tags kind` on ubuntu-latest, exercising the full deploy → helm install → run scenarios → teardown cycle against a fresh KIND cluster on every PR that touches deployment-relevant paths. Wall-clock cost (measured across the prior PR's iterations): - Setup + tool installs: ~30s - Bazel cache warm: ~10s (~60s cold via setup-bazel cache) - bazel build //src/cli:cli: ~30-60s (cached after first run) - KIND cluster + osmo/quick-start helm install: ~3-4 min - 10 scenarios at --jobs=1: ~5-6 min - Cleanup: ~10s - Total: ~11 min warm / ~14 min cold Concurrency: concurrency.cancel-in-progress keeps rapid pushes from queueing. Path filter: only triggers on PRs touching deployments/charts, scripts, values; test/oetf; test/workflow; bzl; MODULE.bazel; src; or this file. This workflow depends on the OETF framework + KIND adapter landing in main. Companion PR opens once that lands. Issue - None ## Checklist - [x] I am familiar with the [Contributing Guidelines](https://github.com/NVIDIA/OSMO/blob/main/CONTRIBUTING.md). - [x] New or existing tests cover these changes. - [x] The documentation is up to date with these changes. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * oetf-kind.yaml: skip deploy step when OETF framework not on tree This workflow exists for PR #1062's framework but lives on its own PR (#1066) so reviewers can evaluate the CI gate separately. Until #1062 merges, //test/oetf:deploy_and_run doesn't exist on main and the deploy step fails red on every push — including pushes against THIS PR itself. Guard via `[ -f test/oetf/BUILD ]` probe + per-step `if:` conditions. When the framework is absent the workflow exits cleanly with a notice; once #1062 lands the probe flips to true on every future PR. * oetf-kind.yaml: add test/{smoke,scenarios} to paths filter Companion #1062 reorganized test/oetf/staging/{smoke,scenarios}/ → top-level test/{smoke,scenarios}/. Add the new paths to the workflow's trigger filter so changes there fire the KIND deploy CI. * oetf-kind.yaml: diagnose KIND state on failure, surface PASS/FAIL in step summary Failure debuggability + at-a-glance PR check UX: * Add "Diagnose KIND state on failure" step that runs only on deploy_and_run failure. Dumps `kubectl get pods -A`, recent events, describe + logs (incl. --previous) for non-Ready pods, helm releases, and disk space. Each subsection is wrapped in `::group::` so reviewers fold what they don't need. All `|| true` — diagnostic must never mask the real failure. * Pass `--keep` + `--keep-on-failure` to deploy_and_run so the cluster survives test-or-deploy failure long enough for the diagnostic step to capture state. The explicit "Cleanup KIND cluster" step (if: always()) still tears it down afterward. * Add "Write run summary to GitHub step summary" step that pulls kind-smoke-result.json and renders the [PASS]/[FAIL] table directly on the PR check page — reviewers see results without scrolling the 10-minute step log. Falls back to a deploy-failure note when JSON is missing. * Wrap the bazel run in `/usr/bin/time -p` with an EXIT trap so total wall clock lands in both the step log and the step summary, success or failure. Surfaces regressions past the ~15-min PR-gate budget. * Add a pre-deploy `df -h /` step so disk-pressure regressions on the ubuntu-latest runner are visible up front instead of as opaque image pull / KIND failures later. * Drop the "Detect OETF framework presence" skip and its `if:` guards on Build osmo CLI / Run OETF deploy_and_run — #1062 has merged, so the framework is always on the tree and the skip is dead code. Run #22 (clean rebase onto current main) passed 10/10 in 11m03s total (10m03s for the deploy_and_run step), confirming the workflow stays inside the 15-min PR-gate budget on a cold ubuntu-latest runner. * oetf-kind.yaml: write kind-smoke-result.json under $GITHUB_WORKSPACE `bazel run` executes the binary with cwd set to the runfiles sandbox, not the workspace root. A relative `--output-json kind-smoke-result.json` landed somewhere under bazel-out where neither actions/upload-artifact nor the "Write run summary" step could find it — both silently fell through (the artifact step warned "No files were found", the summary step hit its deploy-failure fallback even on green runs). Fix by passing an absolute path under $GITHUB_WORKSPACE (the workspace root in GitHub Actions). Confirmed broken in runs #22 and #23 by reading the "Upload OETF result JSON" step log. * oetf-kind.yaml: build PR images locally, narrow test set to smoke + 2 cheap scenarios Without --build-local the gate runs against released nvcr.io/nvidia/osmo:6.2 images and never exercises the PR's src/ changes — any service-code regression slips through green. The paths filter already includes src/**, so the intent was always a real per-PR code gate; this commit makes the behavior match. * Add --build-local. KindAdapter builds the 9 Python service images via bazel oci_load + docker load + kind load, and the web-ui via docker buildx (multi-stage Next.js). The two pipelines run concurrently. The chart picks up the kind-loaded images via global.osmoImageLocation= osmo.local + per-service imagePullPolicy=IfNotPresent overrides that the adapter sets automatically when build_local=True. * Narrow the test set to keep PR-gate cost down: smoke: api-checks (~3s), websocket-checks (~2s) scenarios: templates (~3s), mount-validation (~3s) Smoke covers the service+router+logger HTTP/WS surfaces; templates + mount-validation submit workflows so backend-listener / backend-worker / osmo_ctrl runtime get touched too. The slow scenarios — router- connectivity (140s), serial-workflow-mounting (60s), logger- connectivity (30s) — and the broader negative-path scenarios belong in a separate nightly/manual full run, not the per-PR gate. * Use --target-pattern to scope the bazel query: smoke/... is wildcarded (api-checks + websocket-checks are picked up by --tags kind, cli-checks is dropped because it has no kind tag), templates + mount-validation are listed explicitly via the comma-joined form. * oetf-kind.yaml: bump job timeout to 90m while measuring cold --build-local * oetf-kind.yaml: revert --build-local + 90m timeout; document why Tested --build-local on hosted ubuntu-latest with a narrowed test set (smoke + templates + mount-validation, 4 tests). Result: not viable as-is. Measurements (run #25 cancelled at 60m timeout, run #26 with bumped 90m timeout, both cold): * Run #25 (sha 9d1dd6f, 60m timeout): step 12 hit job timeout while building images. No completion data. Cancelled. * Run #26 (sha bdeb6b9, 90m timeout): step 12 ran 15m15s before failing. Bazel built the 9 OSMO service OCI image tarballs in 7m9s (cold disk- cache hit, repo cache warm from prior runs). docker load + kind load began at 19:43:12; kind-load filled the host disk while distributing backend-worker to the 6-node KIND cluster — `ctr ... images import` failed with `no space left on device` at 19:49:58. Diagnostic step reported disk going from 87 GB free pre-deploy to 11 GB free at failure (76 GB consumed by image tarballs + docker storage + KIND node containerd duplication across 6 nodes). * Run #26 also exposed a separate blocker: web-ui build via docker buildx failed in 1s because test/oetf/local_images.py hardcodes `_UI_SOURCE_RELPATH = "external/src/ui"` — a path that only resolves in the internal-repo layout (which mounts the public repo at external/). In the public NVIDIA/OSMO checkout the source lives at src/ui. Concurrent build kept service builds going so this didn't block the run on its own, but --build-local can never succeed on the public checkout until that path autodetects. Conclusion: for the per-PR gate to ship now, this commit keeps the deploy against released NGC images (verified-working 11m03s avg over runs #22/23/24) and documents what --build-local would need to be viable on hosted GHA runners: - test/oetf/local_images.py UI-path autodetect (upstream fix) - one of: single-node KIND profile that lets the chart still schedule, a runner-disk reclamation step before kind load, or a larger runner (ubuntu-latest-4core / self-hosted) - docker buildx GHA-layer cache for the Next.js web-ui rebuild The 90m timeout is reverted to 60m. Warm-cache --build-local cost remains unmeasured because no cold run completed cleanly. * oetf-kind: enable --build-local with runner-disk reclaim + UI path autodetect Re-enables --build-local on the workflow with the two blockers from the prior measurement attempt addressed: * test/oetf/local_images.py: _ui_dir() now picks the first existing directory from (external/src/ui, src/ui), supporting both layouts: - internal overlay (mounts the public OSMO repo at external/) → external/src/ui (preserves prior behavior) - public standalone checkout (NVIDIA/OSMO GHA runs) → src/ui Falls back to the first candidate if neither exists so the downstream `docker buildx build` produces the same actionable "path not found" error path it always did. * .github/workflows/oetf-kind.yaml: new "Free runner disk for --build-local" step before the deploy reclaims 25-30 GB by removing /usr/share/dotnet, /usr/local/lib/android, /opt/ghc, /opt/hostedtoolcache/PyPy, /opt/hostedtoolcache/CodeQL — pre-installed tooling the OETF gate doesn't use. Prior cold run consumed 76 GB against an 87-GB free baseline and failed at `kind load backend-worker` with no-space-left-on-device. Pairs disk reclaim with --build-local so the 9 OCI tarballs × 6-node containerd duplication has headroom. Workflow also narrows the test set to the smoke + 2 cheap scenarios per the per-PR-gate cost budget: smoke:api-checks + smoke:websocket-checks exercise the service+router+logger HTTP/WS surfaces; scenarios:templates + scenarios:mount-validation submit workflows so backend-listener, backend-worker, and osmo_ctrl runtime get touched. Slow scenarios (router-connectivity 140s, serial-workflow-mounting 60s, logger-connectivity 30s) stay out of the per-PR gate. * test/oetf/local_images: force materialization in build_and_load Project .bazelrc sets `build --remote_download_outputs=minimal`, which makes disk-cache hits leave outputs as reference-only artifacts instead of writing them to bazel-out. The downstream `docker load -i bazel- out/.../tarball.tar` step then fails with `no such file or directory` because the tarball never landed on the local filesystem. Symptom only shows up when the bazel disk cache is warm (e.g. the second --build-local run after a prior cache-save). Cold runs execute the action locally and materialize anyway, so the bug stayed hidden in the original PR #1062 measurements. Fix by adding `--remote_download_outputs=all` to the bazel build inside build_and_load — overrides the project default for this one invocation without touching .bazelrc. Targeted to the OCI image builds where local materialization is a strict requirement. Discovered in run #28 of the local-kind-deployment workflow when the second --build-local attempt (after run #26 saved a warm disk cache) failed all 9 docker loads at 20:31:43 with `no such file or directory`. * oetf-kind.yaml: revert --build-local; disk pressure unfit for hosted runner Three measurement attempts on hosted ubuntu-latest, each failing at a different point, all rooted in the same constraint: Run #26 (sha bdeb6b9): step 13 ran 15m15s. Bazel built 9 service images. kind-load distributed 8/9 to 6 nodes, then failed loading backend-worker — `no space left on device` on KIND-node containerd. Disk: 87 GB free pre-deploy → 11 GB at failure. Run #28 (sha 3c44ac4, after 25-30 GB reclaim): step 13 ran 3m13s. Bazel build was a warm disk-cache hit but tarballs never landed on disk because the project default `--remote_download_outputs=minimal` leaves cache-hit artifacts as references. All 9 docker loads failed with `no such file or directory`. Disk: 100 GB → 86 GB. Run #29 (sha 4ff9641, with materialization fix): step 13 ran 12m08s, then the Actions runner process itself crashed with `No space left on device` writing its own diagnostic log. Cleanup / diagnostic / upload steps never ran (all "pending" in API). Confirms disk-out again, now during kind-load with more images loading further. The 6-node KIND profile (one per chart node_group) means every loaded image is duplicated into 6 separate containerd content stores. 9 services × ~500MB × 6 nodes + web-ui = ~30 GB just for image storage inside KIND nodes, layered onto bazel-out + docker daemon + the runner itself. Reclaim of /usr/share/dotnet, /usr/local/lib/android, /opt/ghc, /opt/hostedtoolcache/{PyPy,CodeQL} adds 25-30 GB of headroom — not enough. Enabling --build-local on hosted GHA needs one of: - single-node KIND profile in the chart (kills the 6x duplication) - larger runner (ubuntu-latest-4core / 8core / self-hosted) - chart-side image pinning that limits which images land on which nodes (more invasive) The two upstream local_images.py bugs uncovered along the way (UI path hardcoded for internal-overlay layout, missing materialization with --remote_download_outputs=minimal) are kept in this PR's accompanying commits — they're correctness bugs regardless of where --build-local runs. This commit reverts the workflow-side change so the per-PR gate ships green against released NGC images today, while the disk-pressure follow-ups are tracked. Measured avg without --build-local: 11m03s (run #22), 10m51s (run #23), 11m06s (run #24) — well inside the 15-min budget. * oetf-kind.yaml: re-enable --build-local with docker storage on /mnt Prior measurement runs (#26, #29) exhausted the ubuntu-latest 145-GB root volume during kind-load; the runner process itself crashed with `No space left on device` in run #29. The 25-30 GB pre-installed- tooling reclaim wasn't enough — 6-node KIND × 9 service images each duplicate into containerd content stores under /var/lib/docker, plus bazel-out, plus docker daemon caches. This commit relocates /var/lib/docker → /mnt/docker before deploy, where ubuntu-latest's separate ~80 GB ephemeral SSD lives. Combined with broader pre-installed-tooling reclaim (now also dotnet, Android SDK, GHC, PyPy cache, CodeQL, boost, chrome, swift, az, microsoft tools, pipx, powershell), the build has effectively 100+ GB of fresh space on /mnt for docker/containerd plus ~30 GB extra on / for bazel. Re-enables --build-local with the same narrowed test set as the prior attempt: smoke + templates + mount-validation only — 4 quick tests that exercise service+router+logger HTTP/WS + workflow submission. This is the third attempt at measuring --build-local end-to-end on hosted GHA. If this still doesn't fit, the conclusion stands: hosted ubuntu-latest needs either single-node KIND or a larger runner. * oetf-kind.yaml: skip web-ui from --build-local for disk headroom Saves ~8 GB across the run: - Next.js docker buildx intermediates: ~5 GB - web-ui bazel-out tarball: ~500 MB - web-ui image × 6 KIND nodes containerd: ~1.8 GB - misc layers + docker daemon cache: ~700 MB Achieved by: - --build-images=service,agent,logger,worker,delayed-job-monitor, router,authz-sidecar,backend-listener,backend-worker (everything except web-ui) - --extra-set web-ui.services.ui.replicas=0 to scale the chart's web-ui Deployment to zero so `kubectl wait --for=condition= Available --all` doesn't hang on a missing image. Smoke tests don't exercise UI endpoints, so this is safe coverage-wise. Combined with the prior `Free runner disk + relocate docker to /mnt` step (~22 GB reclaim), the working budget on hosted ubuntu-latest is ~110 GB free at deploy start — should leave ~10-15 GB headroom over the measured peak consumption. Note: /mnt on ubuntu-latest is the SAME physical volume as / (confirmed in run #31 via `df -h`: both show /dev/root). The relocate step still runs because moving Docker's storage root changes layout, but disk budget is unchanged. Real headroom comes from the reclaim + skip-UI. * oetf-kind.yaml: final revert of --build-local — 5 measurement attempts Final measurement record on hosted ubuntu-latest: Run #26 (sha bdeb6b9): vanilla --build-local cold. 15m15s before kind-load of backend-worker failed with `no space left on device` on KIND-node containerd. Disk 87 GB → 11 GB. Run #28 (sha 3c44ac4): + 22 GB pre-installed tooling reclaim. 3m13s before all 9 `docker load` failed with `no such file or directory`. Root cause: project's `--remote_download_outputs=minimal` left disk-cache-hit tarballs unmaterialized. FIXED in local_images.py. Run #29 (sha 4ff9641): + materialization fix. 12m08s before the GHA runner process itself crashed with `No space left on device` writing its diagnostic log. Cleanup/diagnostic/upload steps never ran (all "pending" in API). Run #31 (sha c61fc78): + relocate /var/lib/docker → /mnt/docker. 12m46s, identical runner crash. Discovery: ubuntu-latest's /mnt is on the SAME physical volume as / (confirmed via df -h: both show /dev/root). Docker relocation didn't add capacity. Run #32 (sha d5398c5): + skip web-ui from build + scale chart's web-ui Deployment to 0 (saves ~8 GB across Next.js intermediates, bazel-out, and 6-node image storage). Same runner crash. Cold measurement: --build-local does not fit on hosted ubuntu-latest with the chart's 6-node KIND profile, even after exhausting reasonable mitigations (disk reclaim, docker storage relocation, skipping web-ui). Each loaded image is duplicated into 6 separate containerd content stores. The math: 9 service images × ~500 MB × 6 nodes overruns the runner's 145-GB volume. Warm measurement: N/A — never achieved a successful cold run, so no warm-cache baseline could be established. Enabling --build-local will need ONE of: - Single-node KIND profile in the chart (kills 6x duplication; requires chart-side accommodation for node_group selectors) - Larger runner (ubuntu-latest-4core, ubuntu-latest-8core, or self-hosted with bigger ephemeral storage) - Per-node image placement so each image only lands on the nodes that schedule it (most invasive) Two real upstream bugs uncovered during measurement are kept in this PR's accompanying commits — both are correctness issues independent of where --build-local runs: - test/oetf/local_images.py _ui_dir() now autodetects internal- overlay vs public-checkout layouts - build_and_load() now forces --remote_download_outputs=all so warm disk-cache hits still write tarballs to bazel-out This commit reverts the workflow-side --build-local change so the per-PR gate ships green against released NGC images today (verified 11m avg, 10/10 PASS). The architectural follow-ups above are tracked for when --build-local against PR-built images becomes viable. * build-local: intra-step host-docker + tarball cleanup; re-enable in CI After kind-load completes for an image, the host's docker daemon copy and the bazel-out tarball are redundant — each KIND node now owns its own containerd copy. Reclaiming both inline lets the run survive on hosted ubuntu-latest's 145 GB volume, where the chart's 6-node KIND profile multiplies image storage by 6x. local_images.build_and_load now, per image: - docker rmi <tag> after kind-load (`|| true`-style, no-op on failure) - os.remove(tarball) for the bazel-out file Combined with the existing workflow-side mitigations (free-disk + narrowed test set + skip web-ui + chart's web-ui replicas=0), this gives enough margin on hosted ubuntu-latest to complete the build → deploy → test cycle end-to-end. Re-enables --build-local in the workflow with the full pipeline. * oetf: --build-local + --use-local-registry path (replaces kind-load 6x duplication) The chart's 6-node KIND profile makes `kind load docker-image` duplicate every loaded image into 6 separate containerd content stores. With 9 service images that's 54 image-copies on disk, which overruns hosted GitHub Actions ubuntu-latest (six measured attempts on this PR all crashed with `No space left on device`). New opt-in path: `oetf:deploy --build-local --use-local-registry` runs a host-side `registry:2` container, retags built images as `localhost:5001/osmo/<svc>:latest-<arch>` and `docker push`es them. KIND nodes pull on-demand via a containerdConfigPatches + per-node `hosts.toml` mapping `localhost:5001` -> `kind-registry:5000`. Only the nodes whose pods actually schedule an image download it, so the disk multiplier drops from 6x to 1-2x. Default `--build-local` (without the new flag) is unchanged — still uses `kind load docker-image`, still works on developer workstations with plenty of disk. Touched files: * test/oetf/cli_args.py — new `--use-local-registry` flag. * test/oetf/local_images.py — - `build_and_push_to_registry()` mirrors `build_and_load()` but docker-pushes to localhost:5001 instead of `kind load`. - `ensure_local_registry()` starts the registry container idempotent. - `connect_registry_to_kind()` joins the kind docker network and writes per-node `/etc/containerd/certs.d/localhost:5001/hosts.toml`. - `patched_kind_config_with_registry()` writes a temp copy of the bundled KIND config with `containerdConfigPatches` appended. * test/oetf/deploy_adapters/kind_adapter.py — `use_local_registry: bool` field; `_create_cluster_if_missing` wires the registry before/after cluster creation when the flag is set. * test/oetf/deploy_adapters/factory.py — propagates the flag and switches the pre-install hook to push-to-registry; chart's `global.osmoImageLocation` switches to `localhost:5001/osmo`. Web-ui still uses `kind load` via `build_and_load_ui` — the workflow skips web-ui from --build-local and scales its chart Deployment to 0 for now (it's not exercised by smoke tests). Registry-push for UI is a follow-up. Workflow uses `--build-local --use-local-registry` along with the disk-reclaim step + narrowed test set (smoke + templates + mount-validation). Verification of cold + warm wall-clock costs is the follow-up to this commit. * oetf-kind: also push web-ui via registry; remove deadlocking replicas=0 Run #35 deadlocked: ingress-nginx (released as the `quick-start` pod under the umbrella chart) has an init container `wait-for-web-ui` that polls `osmo-ui:80`. With `--extra-set web-ui.services.ui.replicas=0`, the osmo-ui Service has no endpoints, so the init never completes → ingress never goes Ready → every downstream pod that has a `wait-for-ingress` init blocks too (backend-listener, backend-worker, config-setup, token job). Drop the replicas=0 override. Build web-ui too, but via the registry- push path (single host-side copy, ~1 GB) instead of `kind load` (~5 GB across 6 nodes). New `build_and_push_ui_to_registry()` mirrors the service push path: docker buildx → docker push → docker rmi host copy. With all 10 images on the registry, disk usage stays in the ~25-30 GB range (well inside the 145 GB volume after reclaim). * ci: trigger warm-run measurement after cold success (no-op) * test/oetf: fix pylint + test_local_images after registry-mode landings Two CI failures uncovered by `//test/oetf:local_images-pylint` and `//test/oetf/tests:test_local_images`: 1. local_images.py — pylint C0301 (line too long, 101 chars) on the `tarball_abs = os.path.join(...)` ternary; broken across three lines with the absolute-path branch first. Pylint C0415 (import outside toplevel) on `import tempfile` inside patched_kind_config_with_ registry; moved to module-level imports. 2. test_local_images.py — build_and_load now does per-image `docker rmi -f <tag>` after kind-load to reclaim host-docker storage (8 subprocess calls per 2 images, not 6). Rewrite the test's total-count assertion as per-operation counts so it accurately describes the contract: 1 bazel build, 1 cquery, 2 docker load, 2 kind load, 2 docker rmi for 2 images. Adds explicit coverage that the cleanup runs once per image. Full //test/oetf/tests:all + //test/oetf:all sweep passes locally (60/60 tests green). * oetf-kind.yaml: address CodeRabbit security suggestions * actions/checkout: pass persist-credentials: false. No git push from this job — strip the auto-injected token from the workspace so the upload-artifact steps (kind-smoke-result, bazel-testlogs) can't carry it into the artifact tarball. * actions/upload-artifact: pin both invocations to the v4.6.1 SHA (4cec3d8aa04e39d1a68397de0c4cd6fb9dce8ec1) instead of the mutable v4 tag, matching the SHA-pinning convention used for actions/checkout and bazel-contrib/setup-bazel elsewhere in the file. --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Checklist