Repository navigation
Spec REMOTE-3452: self-hosted Kubernetes network policies - #169
Draft
warp-agent-staging[bot] wants to merge 1 commit into
Draft
warp-agent-staging[bot] wants to merge 1 commit into
warp-agent-staging[bot] wants to merge 1 commit into
Conversation
Contributor
Author
|
This PR was generated with Warp. Comment    |
bnavetta
reviewed
Oct 18, 2026
| ### 1. Operator configuration and supported boundary | ||
| - Add `backend.kubernetes.supports_network_policies: false`. Wire it through `mergeConfig` to `KubernetesBackendConfig.SupportsNetworkPolicies`. Add Helm `kubernetesBackend.supportsNetworkPolicies: false`. | ||
| - This is a file/Helm option. Do not add a global CLI boolean that can imply support on Docker, direct, or command backends. | ||
| - Add a `network_policy_dns` block under `backend.kubernetes`, with Helm equivalent `kubernetesBackend.networkPolicyDNS`: |
Collaborator
There was a problem hiding this comment.
It's pretty common for the worker to run within the Kubernetes cluster. In that case, can it automatically resolve the cluster's DNS server?
| - When support is enabled, require nonempty, valid resolver addresses and nonempty namespace/labels. These describe the cluster's trusted DNS service. Use them for native DNS grants and Cilium DNS endpoint selection. | ||
| - The native resolver sends lookups to the configured resolver addresses, with bounded UDP/TCP resolution. The task's final DNS configuration must use the same trusted DNS view. Document how to obtain the cluster DNS Service addresses. | ||
| - V1 targets ordinary Pod-network tasks with cluster DNS. NodeLocal DNS, custom DNS forwarding, service meshes, and secondary interfaces require operator validation of equivalent enforcement; the worker must not synthesize broad network grants to accommodate them. | ||
| - Before enabling support, the operator must install and verify a persistent native egress default-deny policy selecting the reserved label `oz-network-restricted=true`. Ship an example manifest and installation instructions in the worker README. Do not create this baseline moments before each task. |
Collaborator
There was a problem hiding this comment.
Let's call it warp-network-restricted instead. We should also put the policy in the Helm chart
| - The operator must verify that the CNI has handled the baseline before enabling support. Kubernetes exposes no portable policy-enforced readiness status. A Helm success or successful policy POST does not satisfy this prerequisite. | ||
| - The baseline must not select unrestricted task Pods. Other native, Cilium, or administrative policies must not widen restricted Pods' egress. Admission must preserve the worker's selector labels and the task's network boundary. | ||
| - Task identities must not create Pods, modify labels/policies, access a privileged daemon, or escape to host networking. The task ServiceAccount remains separate from the worker's policy-management ServiceAccount. | ||
| - Reject restricted templates with `hostNetwork: true`, a privileged container, or added `NET_ADMIN`/`SYS_ADMIN` capabilities. Inspect regular and init containers in the final merged template. Root sidecar-copy init containers remain supported; root UID alone is not privileged mode. |
Collaborator
There was a problem hiding this comment.
If the user chooses to weaken their container config, I don't think we need to block it. We should not set these ourselves though
| - Cluster Service DNAT and Cilium-managed endpoint identity can deny traffic that an IP rule names. Document these provider limits. Never solve reachability by granting an entire cluster/network. Acceptance tests use controlled external destinations and the declared DNS service. | ||
| ### 4. Launch ordering and failure safety | ||
| - Keep nil-policy launches on their existing path. The following state machine applies only to restricted launches. | ||
| - Allocate a fresh UUID token for each created Job. Add worker-owned labels `oz-network-restricted=true` and `oz-network-policy-id=<token>` to its Job and Pod template. Operator extra labels must not override these values. |
Collaborator
There was a problem hiding this comment.
The network policy ID label should also be warp- prefixed. Generally, I think we should reserve that prefix for Warp and not operator labels
| - Cluster Service DNAT and Cilium-managed endpoint identity can deny traffic that an IP rule names. Document these provider limits. Never solve reachability by granting an entire cluster/network. Acceptance tests use controlled external destinations and the declared DNS service. | ||
| ### 4. Launch ordering and failure safety | ||
| - Keep nil-policy launches on their existing path. The following state machine applies only to restricted launches. | ||
| - Allocate a fresh UUID token for each created Job. Add worker-owned labels `oz-network-restricted=true` and `oz-network-policy-id=<token>` to its Job and Pod template. Operator extra labels must not override these values. |
Collaborator
There was a problem hiding this comment.
Instead of a fresh UUID, can we use the run and execution IDs?
| - Create the Job with `suspend: true`, `backoffLimit: 0`, and `restartPolicy: Never`. Its metadata records launch state `prepared`. Capture its UID before policy creation. A suspended Job starts no init, setup, or task containers. | ||
| - Compile and create one provider policy in the task namespace. Select its token label. Store worker/task/execution identity, provider, Job name, and full Job UID in policy labels/annotations. | ||
| - Check the API response matches the intended selector and rules. Admission mutation that broadens the policy fails the launch. Unknown timeouts are ambiguous: re-read by name and verify identity/content before proceeding. | ||
| - Resume only with a UID/resourceVersion-fenced update that changes `prepared` to `launched` and clears suspension. A conflicting or replaced Job is not safe to resume. |
Collaborator
There was a problem hiding this comment.
This doesn't seem necessary since we can create the policy before the job?
| - Never delete the policy in a generic `ExecuteTask` defer. Cleanup is a separate state check. Failure or cancellation can return before Kubernetes has terminated containers. | ||
| ### 5. Policy cleanup and reconciliation | ||
| - Implement policy-only reconciliation at worker startup and every 30 seconds while support is enabled. Scope lists to the configured namespace and worker-pool hash. It does not need server secrets or full run reattachment. | ||
| - Do not make the policy a Job-owned garbage-collection dependent. Background Job deletion could garbage-collect the policy while Pods still terminate. Explicit reconciliation controls deletion order. |
Collaborator
There was a problem hiding this comment.
OOC, is this something Kubernetes supports?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Spec only for REMOTE-3452. No runtime, protocol, or chart implementation is included.
specs/REMOTE-3452/PRODUCT.mdrecords the agreed behavior and exclusions.specs/REMOTE-3452/TECH.mddefines assignment plumbing, Cilium/native enforcement, operator prerequisites, terminal failures, safe cleanup, and executable implementation acceptance checks.--no-cleanup.Validation
git diff --cached --checkpassed before commit.Approval
Proposed specification awaiting Ben Navetta's approval. Approval is not deployment or feature-flag authorization.