Skip to content

Commit d0c6dc3

Browse files
feat(kubernetes): support corporate upstream proxy (#2633)
* feat(kubernetes): support corporate upstream proxy Signed-off-by: loveRhythm1990 <qiuweimin@126.com> * fix(kubernetes): reject proxy_auth_secret_key values Kubernetes cannot create Gateway validation accepted proxy_auth_secret_key values that Kubernetes rejects when creating the Secret (keys longer than 253 bytes, or the reserved "."/".." names), turning an invalid deployment setting into repeated sandbox Pod-provisioning failures instead of a startup error. Reject them in validate_upstream_proxy_config so they fail closed at gateway startup. Signed-off-by: loveRhythm1990 <qiuweimin@126.com> * docs(skill): add corporate upstream proxy checks to debug-openshell-cluster Add a Kubernetes corporate upstream proxy troubleshooting section covering rendered [openshell.drivers.kubernetes] configuration, credential Secret volume events, supervisor arguments and mounts confined to the network- supervising container, and proxy reachability. Signed-off-by: loveRhythm1990 <qiuweimin@126.com> --------- Signed-off-by: loveRhythm1990 <qiuweimin@126.com>
1 parent 7a7b3ee commit d0c6dc3

19 files changed

Lines changed: 1174 additions & 24 deletions

File tree

.agents/skills/debug-openshell-cluster/SKILL.md

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -471,6 +471,56 @@ kubectl -n <sandbox-namespace> logs <sandbox-pod> -c openshell-supervisor-networ
471471
kubectl -n <sandbox-namespace> logs <sandbox-pod> -c agent --tail=200
472472
```
473473

474+
#### Corporate upstream proxy
475+
476+
When the deployment routes sandbox egress through a corporate HTTP forward
477+
proxy, the operator-owned settings render under `[openshell.drivers.kubernetes]`
478+
from the Helm `upstreamProxy` values. Absent proxy configuration preserves
479+
direct-dial egress; any present-but-invalid value fails closed at gateway
480+
startup (`validate_upstream_proxy_config`) rather than silently reverting to a
481+
direct connection. Confirm the rendered configuration first:
482+
483+
```bash
484+
kubectl -n openshell get configmap openshell-config -o jsonpath='{.data.gateway\.toml}' | grep -E 'https_proxy|no_proxy|proxy_auth_secret_(name|key)|proxy_auth_allow_insecure|proxy_connect_by_hostname'
485+
helm -n openshell get values openshell | grep -A8 upstreamProxy
486+
```
487+
488+
Only `http://host:port` forward proxies are supported; `https://` proxy URLs and
489+
plain-HTTP egress are out of scope and rejected. Proxy credentials require
490+
`topology = "sidecar"` — combined topology shares the credential mount with the
491+
workload, so the gateway rejects credentials there. The credential Secret named
492+
by `proxy_auth_secret_name` must exist in the sandbox namespace with the key
493+
named by `proxy_auth_secret_key`, and Kubernetes will not create keys longer
494+
than 253 bytes or named `.`/`..`.
495+
496+
The proxy arguments and credential mount are injected only into the container
497+
that runs network supervision (the `agent` container in combined topology, the
498+
`openshell-supervisor-network` sidecar in sidecar topology). The one-shot
499+
`openshell-network-init` container and the process `agent` container in sidecar
500+
topology must never receive them. The credential is projected read-only as the
501+
`openshell-upstream-proxy-auth` volume at `/run/openshell/upstream-proxy-auth`
502+
and passed as `--upstream-proxy-auth-file`; it must never appear in env,
503+
annotations, or command arguments.
504+
505+
```bash
506+
kubectl -n <sandbox-namespace> get secret <proxy-auth-secret> -o jsonpath='{.data}' >/dev/null && echo "secret present"
507+
kubectl -n <sandbox-namespace> get pod <sandbox-pod> -o jsonpath='{range .spec.containers[*]}{.name}{" "}{.command}{"\n"}{end}' | grep -- '--upstream-'
508+
kubectl -n <sandbox-namespace> get pod <sandbox-pod> -o jsonpath='{range .spec.containers[*]}{.name}{": "}{range .volumeMounts[*]}{.name}{" "}{end}{"\n"}{end}' | grep upstream-proxy-auth
509+
kubectl -n <sandbox-namespace> get events --sort-by=.lastTimestamp | grep -Ei 'secret|MountVolume' | tail -n 20
510+
```
511+
512+
A missing Secret or wrong key leaves the pod stuck with a
513+
`MountVolume.SetUp failed` / `secret ... not found` event. If the pod starts but
514+
egress still fails, the corporate proxy itself is the next suspect: policy-
515+
approved TLS CONNECT requests that time out after policy evaluation usually mean
516+
the proxy URL is unreachable from the sandbox namespace, or a cluster-internal
517+
destination that should be direct is missing from `no_proxy`. Inspect the
518+
network supervisor logs for CONNECT and upstream-proxy decisions:
519+
520+
```bash
521+
kubectl -n <sandbox-namespace> logs <sandbox-pod> -c openshell-supervisor-network --tail=200 | grep -Ei 'upstream|connect|proxy'
522+
```
523+
474524
### Step 7: Check VM-Backed Gateways
475525

476526
Use the VM driver logs and host diagnostics available in the user's environment. Verify:

Cargo.lock

Lines changed: 1 addition & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

architecture/sandbox.md

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -190,6 +190,14 @@ file and builds the `Proxy-Authorization: Basic` header; a credential that is
190190
empty, contains control characters, or is not in `user:pass` form is fatal on
191191
both sides.
192192

193+
For Kubernetes sandboxes, the operator configures a Secret name and key rather
194+
than a gateway-host file path. Kubernetes projects that Secret only into the
195+
container that runs network supervision. Proxy credential Secrets require the
196+
sidecar topology, which gives them a separate container boundary from the
197+
workload. Combined topology is rejected because Kubernetes `fsGroup` volume
198+
permission handling can make a shared credential mount readable by the sandbox
199+
group.
200+
193201
The Basic header travels over the plain-TCP connection to the `http://` proxy,
194202
so it is readable on the network path between sandbox host and proxy.
195203
Configuring `proxy_auth_file` therefore requires the explicit opt-in

crates/openshell-driver-kubernetes/Cargo.toml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,7 @@ miette = { workspace = true }
3737

3838
[dev-dependencies]
3939
temp-env = "0.3"
40+
toml = { workspace = true }
4041

4142
[lints]
4243
workspace = true

0 commit comments

Comments
 (0)