Skip to content

Commit aa7ce2e

Browse files
Merge remote-tracking branch 'origin/main' into test/3954-host-lanes-to-testkit/matthewgrossman
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
2 parents b6fad3c + 87ab3d4 commit aa7ce2e

114 files changed

Lines changed: 7885 additions & 1020 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.agents/skills/helm-dev-environment/SKILL.md‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -445,6 +445,16 @@ public `SandboxStatus` image fields. Resume a pending operation with
445445
`SandboxUpdates.UpdateSandbox`, the current resource version, and no selected
446446
fields, following the documented request-ID semantics.
447447

448+
449+
For the opt-in application listener, build/import matching branch images and run
450+
`mise run e2e:kubernetes:app-listener`; see `e2e/kubernetes/app-listener/README.md`
451+
for image, topology, delivery-mode and takeover inputs. The task owns an isolated
452+
Helm release/namespace, temporary TLS material, in-cluster clients, teardown and
453+
evidence under `results/app-listener/`. It checks policy denial, Pod-IP ingress without app Services, shared-owner takeover, gateway-outage expiry and disabled-listener
454+
gateway compatibility, without discovery or a proxy. CI runs four cases under
455+
`test:e2e` and includes them in `Core E2E result`. Its reusable kind setup installs
456+
pinned Calico only for this lane; other jobs retain kindnet. Local k3s already
457+
enforces policy. The lower-level `verify.py` can target a predeployed release.
448458
The kube e2e wrapper creates only one port-forward, to `svc/openshell`; it no
449459
longer forwards the unauthenticated health listener or runs a `/readyz` e2e
450460
target. `/readyz` remains covered by server unit/integration tests.
@@ -645,3 +655,14 @@ identity and contents, command/environment changes, and request replay.
645655
`OPENSHELL_E2E_UPDATE_ORIGINAL_IMAGE` and `OPENSHELL_E2E_UPDATE_TARGET_IMAGE` can
646656
select mirrored Alpine 3.21 and 3.22 manifests. Defaults use the public registry
647657
cache; keep immutable digests when overriding them.
658+
659+
## App listener fixture
660+
661+
The opt-in `deploy/helm/openshell/ci/values-app-listener.yaml` enables the
662+
supervisor listener at port 9443 without discovery or a proxy. Provision the
663+
operator TLS Secret and an ingress-client NetworkPolicy first. Follow
664+
`e2e/kubernetes/app-listener/README.md` for direct in-cluster verification,
665+
shared-member selection, and the HA follow-up checks. Keep the fixture disabled
666+
in default profiles. Dedicated and shared supervisors expose app ports on their
667+
Pod IPs without app Services. Select the member owning the sandbox; automatic
668+
owner selection belongs to the later discovery API.

‎.agents/skills/release-captain/SKILL.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -219,7 +219,7 @@ git worktree remove --force "$wt"
219219
| Transient infrastructure failure in a pre-release, including a registry push | After approval, `gh run rerun <run-id> --failed`. `Release` won't select the run until all of it passes. |
220220
| Pre-release run ends in `startup_failure` | The workflow file is invalid at that commit. Fix it in a new `-pre.N` (Monday step 3). |
221221
| Code defect or `Cargo Deny` advisory | Rerunning won't help. Fix it in a new `-pre.N` (Monday step 3). |
222-
| `Release` fails after selection | After approval, rerun only the failed jobs that haven't published. |
222+
| `Release` fails after selection | After approval, rerun only the failed jobs that haven't published. If the run's artifacts expired or its jobs were skipped, dispatch `Release` again with `-f tag=vX.Y.Z`; it publishes only what is missing. |
223223

224224
## Related
225225

‎.agents/skills/release-openshell/SKILL.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -32,7 +32,9 @@ passed but publication failed, rerun the failed jobs before promoting the candid
3232
Release creates `vX.Y.Z` at that exact commit, then rebuilds and publishes all
3333
stable artifacts without rerunning qualification. Embedded versions change, so
3434
stable artifacts are not byte-identical to the qualified prerelease. An existing
35-
stable tag may be retried only if it already points to the selected commit.
35+
stable tag may be retried only if it already points to the selected commit. A retry
36+
publishes only missing artifacts; published images, charts, release assets, npm,
37+
and PyPI versions are kept, while the snap is uploaded again.
3638
Stable images are saved as OCI archives during the build and pushed under the
3739
stable version tags during publication. Their SBOMs are verified before updating
3840
`latest`; the qualified prerelease's commit-SHA image tags remain unchanged.

‎.agents/skills/test-release-canary/SKILL.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -95,7 +95,7 @@ Note `install.sh` is pulled from `raw.githubusercontent.com/NVIDIA/OpenShell/${h
9595

9696
## Testing artifacts from a specific SHA
9797

98-
`Release Dev` publishes two chart versions for every dev build (see `.github/actions/release-helm-oci/action.yml:89-102`):
98+
`Release Dev` publishes two chart versions for every dev build (see the push steps in `.github/actions/release-helm-oci/action.yml`):
9999

100100
- `oci://ghcr.io/nvidia/openshell/helm-chart:0.0.0-dev` — floating, overwritten on every main push.
101101
- `oci://ghcr.io/nvidia/openshell/helm-chart:0.0.0-dev.<sha>` — immutable, `appVersion` set to the same SHA so it pulls the matching `gateway`, `sandbox`, and `supervisor` images.

‎.agents/skills/watch-github-actions/SKILL.md‎

Lines changed: 9 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -125,6 +125,12 @@ gh run list --json databaseId,status,headBranch,url --jq '.[] | {id: .databaseId
125125

126126
## View Job Logs
127127

128+
For Kubernetes app-listener E2E failures during kind setup, inspect the uploaded
129+
`results/app-listener/cni-setup/` evidence. It includes node and Calico pod
130+
descriptions, events, and current/previous container logs. A CNI rollout timeout
131+
occurs before the listener tests run; use these diagnostics to distinguish image
132+
pulls, init failures, and readiness failures.
133+
128134
For tmachine Kubernetes conformance failures, inspect the `e2e-results-conformance-conformance-ubuntu-k3s-*` artifact. Its `diagnostics/` holds the K3s report with Agent Sandbox controller state, leader leases, events, and current and previous controller logs. The k3s installers check reconciliation after guest boot with a temporary scheduling-gated Sandbox probe, so a failed probe fails provisioning before any test runs. Distinguish a failed controller probe from a failed conformance scenario; gateway connectivity and controller Deployment availability do not prove reconciliation works.
129135

130136
For the `e2e-suites` lane (`.github/workflows/e2e-suites.yml`), the last lines of the job log give the runner's result: PASS, FAIL (exit 1) after the end of the nextest log (which names the failed tests), or ERROR (exit 2) with the step that broke. FAIL and ERROR also appear as annotations on the run's summary page. Then inspect the `e2e-results-<family>-<package>-<environment>-<installer>` artifact, which holds `junit.xml`, `nextest.log` and, after a failure, the guest's `diagnostics/`.
@@ -197,7 +203,7 @@ qualification; stable publication is blocked, while published pre-release
197203
artifacts remain available. The Pre-release failure notification includes the
198204
run link. Both callers share `release-build.yml` and `release-publish.yml`.
199205
Qualification evidence remains in Actions; publishing a copy to OCI storage is best effort.
200-
Snap Store uploads happen only for stable releases after qualification passes.
206+
Pre-release uploads snaps to `latest/beta`; stable releases upload to `latest/stable`.
201207

202208
For a release notes preview, dispatch `release.yml` with `notes-only=true` and
203209
`tag=<stable-tag>`. The generator resolves the previous reachable stable tag;
@@ -214,8 +220,8 @@ and qualification, and keeps the notes only in the `release-notes` artifact.
214220
This job is advisory: a notes failure emits a warning without failing the
215221
prerelease run or blocking promotion. Stable publication reuses the promoted
216222
prerelease's `release-notes` artifact, retagged, and generates notes only when
217-
that preview is unavailable. The `Select Qualified Prerelease` summary says
218-
which happened.
223+
that preview is unavailable. The Release workflow's `Release Notes` job summary
224+
says which happened, and Publish waits for that job.
219225

220226
For `Codex Compatibility Review`, inspect its job summary and the
221227
`compatibility-review-run-<run-id>-attempt-<attempt>` artifact. Confirm the

‎.github/actions/release-helm-oci/action.yml‎

Lines changed: 22 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -3,8 +3,8 @@
33

44
name: Release Helm OCI
55
description: >
6-
Patch chart version/appVersion, refuse duplicate OCI versions on public
7-
releases, package the gateway and workspace charts, and push them to GHCR OCI.
6+
Patch chart version/appVersion, package the gateway and workspace charts, and
7+
push them to GHCR OCI. Public releases never overwrite published versions.
88
99
inputs:
1010
chart-version:
@@ -63,22 +63,38 @@ runs:
6363
echo "workspace_chart_dir=${WORKSPACE_CHART_DIR}" >> "$GITHUB_OUTPUT"
6464
echo "chart_version=${CHART_VERSION}" >> "$GITHUB_OUTPUT"
6565
66-
- name: Refuse duplicate chart version
66+
# Public chart versions are never overwritten. A rerun keeps both published
67+
# charts; a partial publication still fails for manual repair.
68+
- name: Check for published chart version
69+
id: published
6770
if: inputs.release-kind == 'public'
6871
env:
6972
CHART_VERSION: ${{ steps.prep.outputs.chart_version }}
7073
shell: bash
7174
run: |
7275
set -euo pipefail
76+
found=0
7377
for chart in helm-chart openshell-workspace; do
7478
OCI_CHART="oci://ghcr.io/nvidia/openshell/${chart}"
75-
if helm show chart "${OCI_CHART}" --version "${CHART_VERSION}" >/dev/null 2>&1; then
76-
echo "::error::Chart ${chart}:${CHART_VERSION} is already published. Use a new tag or delete the existing package first."
79+
# Only "not found" means unpublished; any other lookup error fails
80+
# rather than risk overwriting.
81+
if lookup_error=$(helm show chart "${OCI_CHART}" --version "${CHART_VERSION}" 2>&1 >/dev/null); then
82+
found=$((found + 1))
83+
elif [[ "${lookup_error}" != *"ghcr.io/nvidia/openshell/${chart}:${CHART_VERSION}: not found"* ]]; then
84+
echo "::error::Could not check whether ${chart}:${CHART_VERSION} is published: ${lookup_error}"
7785
exit 1
7886
fi
7987
done
88+
if [ "${found}" -eq 2 ]; then
89+
echo "Charts ${CHART_VERSION} are already published; keeping them."
90+
echo "published=true" >> "$GITHUB_OUTPUT"
91+
elif [ "${found}" -ne 0 ]; then
92+
echo "::error::Only one chart is published at ${CHART_VERSION}. Delete it or publish the other chart manually."
93+
exit 1
94+
fi
8095
8196
- name: Package Helm chart
97+
if: steps.published.outputs.published != 'true'
8298
env:
8399
GATEWAY_CHART_DIR: ${{ steps.prep.outputs.gateway_chart_dir }}
84100
WORKSPACE_CHART_DIR: ${{ steps.prep.outputs.workspace_chart_dir }}
@@ -91,6 +107,7 @@ runs:
91107
ls /tmp/helm-charts/*.tgz
92108
93109
- name: Push Helm chart to GHCR OCI
110+
if: steps.published.outputs.published != 'true'
94111
shell: bash
95112
run: |
96113
set -euo pipefail

‎.github/actions/setup-e2e-kind/action.yml‎

Lines changed: 56 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,10 @@ inputs:
2929
description: mise release to install
3030
required: false
3131
default: v2026.9.9
32+
enforce-network-policy:
33+
description: Replace kindnet with pinned Calico for NetworkPolicy assertions
34+
required: false
35+
default: "false"
3236

3337
runs:
3438
using: composite
@@ -45,11 +49,62 @@ runs:
4549
REGISTRY_PASSWORD: ${{ inputs.registry-password }}
4650
run: echo "$REGISTRY_PASSWORD" | docker login "${REGISTRY%%/*}" -u "$REGISTRY_USERNAME" --password-stdin
4751

52+
- name: Configure policy-enforcing kind network
53+
id: kind-config
54+
if: inputs.enforce-network-policy == 'true'
55+
shell: bash
56+
run: |
57+
set -euo pipefail
58+
config="$RUNNER_TEMP/kind-network-policy.yaml"
59+
cat > "$config" <<'YAML'
60+
kind: Cluster
61+
apiVersion: kind.x-k8s.io/v1alpha4
62+
networking:
63+
disableDefaultCNI: true
64+
podSubnet: 10.244.0.0/16
65+
YAML
66+
echo "path=$config" >> "$GITHUB_OUTPUT"
67+
4868
- name: Create kind cluster
4969
uses: helm/kind-action@ef37e7f390d99f746eb8b610417061a60e82a6cc # v1.14.0
5070
with:
5171
cluster_name: ${{ inputs.cluster-name }}
52-
wait: 120s
72+
# Nodes cannot become Ready until the following CNI installation.
73+
wait: ${{ inputs.enforce-network-policy == 'true' && '0s' || '120s' }}
74+
config: ${{ steps.kind-config.outputs.path }}
75+
76+
- name: Install policy-enforcing CNI
77+
if: inputs.enforce-network-policy == 'true'
78+
shell: bash
79+
env:
80+
CLUSTER_NAME: ${{ inputs.cluster-name }}
81+
run: |
82+
set -euo pipefail
83+
collect_cni_diagnostics() {
84+
failure_status=$?
85+
set +e
86+
diagnostics="$GITHUB_WORKSPACE/results/app-listener/cni-setup"
87+
mkdir -p "$diagnostics"
88+
kubectl --context "kind-$CLUSTER_NAME" get nodes -o wide > "$diagnostics/nodes.txt" 2>&1
89+
kubectl --context "kind-$CLUSTER_NAME" describe nodes > "$diagnostics/node-details.txt" 2>&1
90+
kubectl --context "kind-$CLUSTER_NAME" -n kube-system get pods -o wide > "$diagnostics/pods.txt" 2>&1
91+
kubectl --context "kind-$CLUSTER_NAME" -n kube-system get events --sort-by=.lastTimestamp > "$diagnostics/events.txt" 2>&1
92+
for component in calico-node calico-kube-controllers; do
93+
kubectl --context "kind-$CLUSTER_NAME" -n kube-system describe pods -l "k8s-app=$component" > "$diagnostics/$component-details.txt" 2>&1
94+
kubectl --context "kind-$CLUSTER_NAME" -n kube-system logs -l "k8s-app=$component" --all-containers --prefix --tail=200 > "$diagnostics/$component.log" 2>&1
95+
kubectl --context "kind-$CLUSTER_NAME" -n kube-system logs -l "k8s-app=$component" --all-containers --prefix --tail=200 --previous > "$diagnostics/$component-previous.log" 2>&1
96+
done
97+
cat "$diagnostics/pods.txt" "$diagnostics/events.txt"
98+
exit "$failure_status"
99+
}
100+
trap collect_cni_diagnostics ERR
101+
curl -fsSL https://raw.githubusercontent.com/projectcalico/calico/v3.30.3/manifests/calico.yaml \
102+
-o "$RUNNER_TEMP/calico.yaml"
103+
sed 's@192.168.0.0/16@10.244.0.0/16@g' "$RUNNER_TEMP/calico.yaml" > "$RUNNER_TEMP/calico-kind.yaml"
104+
kubectl --context "kind-$CLUSTER_NAME" apply -f "$RUNNER_TEMP/calico-kind.yaml"
105+
kubectl --context "kind-$CLUSTER_NAME" -n kube-system rollout status daemonset/calico-node --timeout=300s
106+
kubectl --context "kind-$CLUSTER_NAME" -n kube-system rollout status deployment/calico-kube-controllers --timeout=300s
107+
kubectl --context "kind-$CLUSTER_NAME" wait nodes --all --for=condition=Ready --timeout=120s
53108
54109
- name: Export kind kubeconfig to mise path
55110
shell: bash

‎.github/workflows/branch-e2e.yml‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -459,6 +459,43 @@ jobs:
459459
agent-sandbox-version: ${{ matrix.agent_sandbox_version }}
460460
test-name: ${{ matrix.test_name }}
461461

462+
kubernetes-app-listener-e2e:
463+
needs: [pr_metadata, build-binaries, build-images]
464+
if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_core_e2e == 'true'
465+
strategy:
466+
fail-fast: false
467+
matrix:
468+
include:
469+
- topology: dedicated
470+
delivery: push
471+
takeover: graceful
472+
name: dedicated push
473+
- topology: dedicated
474+
delivery: poll
475+
takeover: graceful
476+
name: dedicated poll
477+
- topology: shared
478+
delivery: push
479+
takeover: graceful
480+
name: shared graceful takeover
481+
- topology: shared
482+
delivery: push
483+
takeover: abrupt
484+
name: shared abrupt takeover
485+
permissions:
486+
actions: read
487+
contents: read
488+
packages: read
489+
uses: ./.github/workflows/e2e-kubernetes-test.yml
490+
with:
491+
image-tag: ${{ github.sha }}
492+
job-name: Kubernetes app listener (${{ matrix.name }})
493+
e2e-task: e2e:kubernetes:app-listener
494+
enforce-network-policy: true
495+
listener-topology: ${{ matrix.topology }}
496+
listener-delivery: ${{ matrix.delivery }}
497+
listener-takeover: ${{ matrix.takeover }}
498+
462499
kubernetes-workspace-managed-e2e:
463500
needs: [pr_metadata, build-binaries, build-images]
464501
if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_core_e2e == 'true'
@@ -548,6 +585,7 @@ jobs:
548585
- podman-external-driver-e2e
549586
- vm-external-driver-e2e
550587
- kubernetes-e2e
588+
- kubernetes-app-listener-e2e
551589
- kubernetes-external-driver-e2e
552590
- kubernetes-workspace-managed-e2e
553591
- kubernetes-workspace-operator-e2e

‎.github/workflows/e2e-kubernetes-test.yml‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -77,6 +77,23 @@ on:
7777
required: false
7878
type: string
7979
default: "gateway sandbox supervisor"
80+
enforce-network-policy:
81+
description: "Install a policy-enforcing CNI for listener isolation checks"
82+
required: false
83+
type: boolean
84+
default: false
85+
listener-topology:
86+
required: false
87+
type: string
88+
default: "dedicated"
89+
listener-delivery:
90+
required: false
91+
type: string
92+
default: "push"
93+
listener-takeover:
94+
required: false
95+
type: string
96+
default: "graceful"
8097

8198
permissions:
8299
actions: read
@@ -129,6 +146,14 @@ jobs:
129146
registry-username: ${{ github.actor }}
130147
registry-password: ${{ secrets.GITHUB_TOKEN }}
131148
mise-version: ${{ inputs.mise-version }}
149+
enforce-network-policy: ${{ inputs.enforce-network-policy }}
150+
151+
- name: Preload listener workload and client fixture
152+
if: inputs.e2e-task == 'e2e:kubernetes:app-listener'
153+
run: |
154+
docker pull --platform linux/amd64 python:3.12-slim
155+
docker image save --platform linux/amd64 --output "$RUNNER_TEMP/listener-python.tar" python:3.12-slim
156+
kind load image-archive "$RUNNER_TEMP/listener-python.tar" --name "$KIND_CLUSTER_NAME"
132157
133158
- name: Run Kubernetes E2E
134159
env:
@@ -142,4 +167,15 @@ jobs:
142167
IMAGE_TAG: ${{ inputs.image-tag }}
143168
OPENSHELL_REGISTRY: ghcr.io/nvidia/openshell
144169
E2E_TASK: ${{ inputs.e2e-task }}
170+
OPENSHELL_E2E_APP_TOPOLOGY: ${{ inputs.listener-topology }}
171+
OPENSHELL_E2E_APP_DELIVERY: ${{ inputs.listener-delivery }}
172+
OPENSHELL_E2E_APP_TAKEOVER: ${{ inputs.listener-takeover }}
145173
run: mise run --no-deps --skip-deps "$E2E_TASK"
174+
175+
- name: Upload listener test evidence
176+
if: always() && inputs.e2e-task == 'e2e:kubernetes:app-listener'
177+
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
178+
with:
179+
name: listener-${{ inputs.listener-topology }}-${{ inputs.listener-delivery }}-${{ inputs.listener-takeover }}
180+
path: results/app-listener/
181+
if-no-files-found: warn

0 commit comments

Comments
 (0)