Skip to content

Commit fc8d0d7

Browse files
test(tmachine): add a K3s HA conformance lane with PostgreSQL and mTLS (#4081)
* test(tmachine): add K3s PostgreSQL and mTLS conformance installer Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * fix(tmachine): reach the HA TLS gateway through its ClusterIP Drop the port-forward restart removed by the ClusterIP change, resolve the Service DNS name covered by the server certificate to the ClusterIP, bound each authenticated API wait attempt, and collect diagnostics from every gateway Pod. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * ci(tmachine): run K3s PostgreSQL and mTLS conformance Add the k3s-ha-tls installer to every core conformance matrix so Branch E2E, Release Dev, Release Tag, and manual Integration Tests cover three gateway replicas sharing PostgreSQL behind mTLS. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * refactor(tmachine): rename k3s-ha-tls installer to k3s-ha TLS is part of the supported HA configuration once gateway replicas share PostgreSQL, so the installer name only needs to distinguish HA from the single-replica k3s lane. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * refactor(tmachine): install K3s HA in a single Helm rollout Add an optional Helm values overlay to the K3s installer so the k3s-ha installer stages PostgreSQL first and installs the three-replica TLS gateway directly, instead of installing a plaintext gateway and upgrading. Drop the redundant unauthenticated-users override and the install-time API wait, which the conformance playbook already performs. Share K3s installer inputs in the tmachine configuration. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * fix(tmachine): make K3s HA staging rerunnable and wait for PostgreSQL Apply the OpenShell namespace instead of creating it so the staging play tolerates an existing namespace, and wait for the PostgreSQL rollout so gateway Pods do not start crash-looping against a missing database. Reflow and pluralize the K3s lane notes in CI.md. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> --------- Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
1 parent 6b7a00e commit fc8d0d7

10 files changed

Lines changed: 151 additions & 16 deletions

File tree

‎.github/workflows/branch-e2e.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -256,6 +256,7 @@ jobs:
256256
[
257257
{"environment":"ubuntu-docker-rootful","installer":"deb","testsuite":"conformance"},
258258
{"environment":"ubuntu-k3s","installer":"k3s","testsuite":"conformance"},
259+
{"environment":"ubuntu-k3s","installer":"k3s-ha","testsuite":"conformance"},
259260
{"environment":"fedora-podman-rootful","installer":"rpm","testsuite":"conformance"},
260261
{"environment":"fedora-podman-rootless","installer":"rpm","testsuite":"conformance"}
261262
]

‎.github/workflows/integration-runner.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,7 @@ on:
2626
[
2727
{"environment":"ubuntu-docker-rootful","installer":"deb","testsuite":"conformance"},
2828
{"environment":"ubuntu-k3s","installer":"k3s","testsuite":"conformance"},
29+
{"environment":"ubuntu-k3s","installer":"k3s-ha","testsuite":"conformance"},
2930
{"environment":"fedora-podman-rootful","installer":"rpm","testsuite":"conformance"},
3031
{"environment":"fedora-podman-rootless","installer":"rpm","testsuite":"conformance"}
3132
]

‎.github/workflows/integration-test.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,7 @@ on:
2929
[
3030
{"environment":"ubuntu-docker-rootful","installer":"deb","testsuite":"conformance"},
3131
{"environment":"ubuntu-k3s","installer":"k3s","testsuite":"conformance"},
32+
{"environment":"ubuntu-k3s","installer":"k3s-ha","testsuite":"conformance"},
3233
{"environment":"fedora-podman-rootful","installer":"rpm","testsuite":"conformance"},
3334
{"environment":"fedora-podman-rootless","installer":"rpm","testsuite":"conformance"}
3435
]

‎.github/workflows/release-dev.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -127,6 +127,7 @@ jobs:
127127
[
128128
{"environment":"ubuntu-docker-rootful","installer":"deb","testsuite":"conformance"},
129129
{"environment":"ubuntu-k3s","installer":"k3s","testsuite":"conformance"},
130+
{"environment":"ubuntu-k3s","installer":"k3s-ha","testsuite":"conformance"},
130131
{"environment":"fedora-podman-rootful","installer":"rpm","testsuite":"conformance"},
131132
{"environment":"fedora-podman-rootless","installer":"rpm","testsuite":"conformance"}
132133
]

‎.github/workflows/release-tag.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -178,6 +178,7 @@ jobs:
178178
[
179179
{"environment":"ubuntu-docker-rootful","installer":"deb","testsuite":"conformance"},
180180
{"environment":"ubuntu-k3s","installer":"k3s","testsuite":"conformance"},
181+
{"environment":"ubuntu-k3s","installer":"k3s-ha","testsuite":"conformance"},
181182
{"environment":"fedora-podman-rootful","installer":"rpm","testsuite":"conformance"},
182183
{"environment":"fedora-podman-rootless","installer":"rpm","testsuite":"conformance"}
183184
]

‎CI.md‎

Lines changed: 9 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -144,16 +144,18 @@ push-triggered workflow jobs themselves.
144144

145145
### K3s conformance version baseline
146146

147-
The tmachine `ubuntu-k3s` conformance lane pins Agent Sandbox v0.5.0 as the
148-
compatibility baseline for the v1beta1 Sandbox API. It does not track the local
147+
The tmachine `ubuntu-k3s` conformance lanes pin Agent Sandbox v0.5.0 as the
148+
compatibility baseline for the v1beta1 Sandbox API. They do not track the local
149149
K3s development default, currently v1.0.3. OpenShell also supports v0.4.6 through
150150
its v1alpha1 fallback, so v0.5.0 is not the overall minimum supported version.
151151

152-
The `ubuntu-k3s` lane registers the gateway's ClusterIP Service directly, so
153-
recreating the Service requires reprovisioning the fixture. Before conformance,
154-
every installer waits up to five minutes for a connected gateway; interactive
155-
shells skip the wait. K3s failures upload diagnostics as `tmachine-diagnostics-*`
156-
artifacts.
152+
The `ubuntu-k3s` lanes register the gateway's ClusterIP Service directly, so
153+
recreating the Service requires reprovisioning the fixture. The `k3s` lane runs
154+
one plaintext gateway. The `k3s-ha` lane runs the supported HA configuration:
155+
three gateway replicas sharing PostgreSQL, serving TLS, with the CLI connecting
156+
over mTLS. Before conformance, every installer waits up to five minutes for a
157+
connected gateway; interactive shells skip the wait. K3s failures upload
158+
diagnostics as `tmachine-diagnostics-*` artifacts.
157159

158160
### Run only the policy advisor conformance tests
159161

Lines changed: 106 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,106 @@
1+
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2+
# SPDX-License-Identifier: Apache-2.0
3+
4+
---
5+
- name: Stage PostgreSQL for the HA gateway
6+
hosts: all
7+
become: true
8+
gather_facts: false
9+
tasks:
10+
- name: Wait for SSH
11+
ansible.builtin.wait_for_connection:
12+
13+
# The install layer boots fresh, so the K3s API may still be starting.
14+
- name: Apply OpenShell namespace
15+
ansible.builtin.command:
16+
argv: [/usr/local/bin/k3s, kubectl, apply, --filename, "-"]
17+
stdin: |
18+
apiVersion: v1
19+
kind: Namespace
20+
metadata:
21+
name: openshell
22+
register: k3s_namespace
23+
until: k3s_namespace.rc == 0
24+
retries: 30
25+
delay: 2
26+
27+
- name: Apply PostgreSQL fixture
28+
ansible.builtin.command:
29+
argv: [/usr/local/bin/k3s, kubectl, --namespace, openshell, apply, --filename, "-"]
30+
stdin: "{{ lookup('ansible.builtin.file', openshell_postgres_fixture) }}"
31+
32+
# Gateway Pods that start before the database crash-loop with growing backoff.
33+
- name: Wait for PostgreSQL
34+
ansible.builtin.command:
35+
argv:
36+
- /usr/local/bin/k3s
37+
- kubectl
38+
- --namespace
39+
- openshell
40+
- rollout
41+
- status
42+
- deployment/openshell-e2e-postgres
43+
- --timeout=300s
44+
changed_when: false
45+
46+
- import_playbook: openshell-k3s.yaml
47+
vars:
48+
openshell_k3s_helm_values_overlay: |
49+
replicaCount: 3
50+
workload:
51+
kind: deployment
52+
server:
53+
externalDbSecret: openshell-e2e-postgres-credentials
54+
disableTls: false
55+
56+
- name: Register the HA gateway over mTLS
57+
hosts: all
58+
become: true
59+
gather_facts: false
60+
tasks:
61+
# The chart's server certificate covers the Service DNS name but not its
62+
# ClusterIP, and the node resolver cannot reach cluster DNS.
63+
- name: Resolve the gateway Service name to its ClusterIP
64+
ansible.builtin.lineinfile:
65+
path: /etc/hosts
66+
regexp: '\sopenshell\.openshell\.svc\.cluster\.local$'
67+
line: "{{ k3s_gateway_address.stdout.split(':')[0] }} openshell.openshell.svc.cluster.local"
68+
69+
- name: Read chart client TLS Secret
70+
ansible.builtin.command:
71+
argv: [/usr/local/bin/k3s, kubectl, --namespace, openshell, get, secret, openshell-client-tls, --output, json]
72+
register: k3s_client_tls
73+
changed_when: false
74+
no_log: true
75+
76+
- name: Create guest mTLS directory
77+
ansible.builtin.file:
78+
path: /home/tmachine/.config/openshell/gateways/tmachine/mtls
79+
state: directory
80+
owner: tmachine
81+
group: tmachine
82+
mode: "0700"
83+
84+
- name: Write guest client TLS bundle
85+
ansible.builtin.copy:
86+
content: "{{ (k3s_client_tls.stdout | from_json).data[item] | b64decode }}"
87+
dest: "/home/tmachine/.config/openshell/gateways/tmachine/mtls/{{ item }}"
88+
owner: tmachine
89+
group: tmachine
90+
mode: "0600"
91+
loop: [ca.crt, tls.crt, tls.key]
92+
no_log: true
93+
94+
# The base playbook registered a plaintext endpoint; replace it.
95+
- name: Remove baseline guest gateway registration
96+
become: false
97+
ansible.builtin.command:
98+
argv: [/usr/local/bin/openshell, gateway, remove, tmachine]
99+
100+
- name: Register guest gateway
101+
ansible.builtin.include_role:
102+
name: openshell_client
103+
apply:
104+
become: false
105+
vars:
106+
openshell_client_gateway_endpoint: "https://openshell.openshell.svc.cluster.local:{{ k3s_gateway_address.stdout.split(':')[1] }}"

‎tests/ansible/playbooks/openshell-k3s.yaml‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -87,6 +87,12 @@
8787
disableTls: true
8888
telemetryEnabled: false
8989
90+
- name: Write OpenShell Helm values overlay
91+
ansible.builtin.copy:
92+
dest: /var/lib/openshell/artifacts/values-overlay.yaml
93+
mode: "0600"
94+
content: "{{ openshell_k3s_helm_values_overlay | default('{}') }}"
95+
9096
- name: Install Agent Sandbox
9197
ansible.builtin.command:
9298
argv:
@@ -132,6 +138,8 @@
132138
- --create-namespace
133139
- --values
134140
- /var/lib/openshell/artifacts/values.yaml
141+
- --values
142+
- /var/lib/openshell/artifacts/values-overlay.yaml
135143
- --wait
136144
- --timeout=5m
137145
environment:

‎tests/ansible/roles/openshell_diagnostics/files/collect-k3s.sh‎

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -43,8 +43,13 @@ collect() {
4343
collect 'Endpoint readiness' /usr/local/bin/k3s kubectl --request-timeout=5s -n openshell get endpointslices \
4444
-o 'custom-columns=NAME:.metadata.name,PORTS:.ports,ADDRESSES:.endpoints[*].addresses,CONDITIONS:.endpoints[*].conditions'
4545
collect 'Gateway events' /usr/local/bin/k3s kubectl --request-timeout=5s -n openshell get events --sort-by=.lastTimestamp
46-
collect 'Gateway current logs' /usr/local/bin/k3s kubectl --request-timeout=5s -n openshell logs openshell-0 -c openshell-gateway --tail=300 --timestamps
47-
collect 'Gateway previous logs' /usr/local/bin/k3s kubectl --request-timeout=5s -n openshell logs openshell-0 -c openshell-gateway --previous --tail=300 --timestamps
46+
# Gateway Pods are named by a StatefulSet or a Deployment, depending on the installer.
47+
mapfile -t gateway_pods < <(timeout 10s /usr/local/bin/k3s kubectl --request-timeout=5s -n openshell get pods \
48+
--selector app.kubernetes.io/name=openshell,app.kubernetes.io/instance=openshell --output name 2>/dev/null)
49+
for pod in "${gateway_pods[@]}"; do
50+
collect "Gateway current logs ($pod)" /usr/local/bin/k3s kubectl --request-timeout=5s -n openshell logs "$pod" -c openshell-gateway --tail=300 --timestamps
51+
collect "Gateway previous logs ($pod)" /usr/local/bin/k3s kubectl --request-timeout=5s -n openshell logs "$pod" -c openshell-gateway --previous --tail=300 --timestamps
52+
done
4853
collect 'Memory' free -m
4954
collect 'Disk' df -h / /var/lib/rancher/k3s
5055
} | sed -E \

‎tests/config.nix‎

Lines changed: 16 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,14 @@ let
1919
OVMF = firmwarePkgs.OVMF;
2020
};
2121
qemu = qemuPkgs.qemu.override { hostCpuOnly = true; };
22+
k3sInputs = {
23+
agent_sandbox_version = "0.5.0";
24+
openshell_cli_binary = "../artifacts/binaries/${muslTarget}/openshell";
25+
openshell_gateway_image = "../artifacts/images/openshell-gateway-tmachine.tar";
26+
openshell_helm_chart = "../artifacts/helm/helm-chart-0.0.0.tgz";
27+
openshell_sandbox_image = "../artifacts/images/openshell-sandbox-tmachine.tar";
28+
openshell_supervisor_image = "../artifacts/images/openshell-supervisor-tmachine.tar";
29+
};
2230
config = (pkgs.formats.yaml { }).generate "tmachine-config.yaml" {
2331
machines = [
2432
{
@@ -85,13 +93,14 @@ let
8593
name = "k3s";
8694
use_galaxy = false;
8795
playbooks = [ "ansible/playbooks/openshell-k3s.yaml" ];
88-
inputs = {
89-
agent_sandbox_version = "0.5.0";
90-
openshell_cli_binary = "../artifacts/binaries/${muslTarget}/openshell";
91-
openshell_gateway_image = "../artifacts/images/openshell-gateway-tmachine.tar";
92-
openshell_helm_chart = "../artifacts/helm/helm-chart-0.0.0.tgz";
93-
openshell_sandbox_image = "../artifacts/images/openshell-sandbox-tmachine.tar";
94-
openshell_supervisor_image = "../artifacts/images/openshell-supervisor-tmachine.tar";
96+
inputs = k3sInputs;
97+
}
98+
{
99+
name = "k3s-ha";
100+
use_galaxy = false;
101+
playbooks = [ "ansible/playbooks/openshell-k3s-ha.yaml" ];
102+
inputs = k3sInputs // {
103+
openshell_postgres_fixture = "../e2e/kubernetes/postgres-fixture.yaml";
95104
};
96105
}
97106
{

0 commit comments

Comments
 (0)