Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion .agents/skills/test-release-canary/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -109,7 +109,6 @@ helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart \
--namespace openshell --create-namespace \
--set server.disableTls=true \
--set server.telemetryEnabled=false \
--set supervisor.sandboxRuntime.networkPolicyEnforced=true \
--wait --timeout 5m

kubectl wait --namespace openshell \
Expand Down
18 changes: 18 additions & 0 deletions .github/copy-pr-bot.yaml
Original file line number Diff line number Diff line change
@@ -1,3 +1,21 @@
enabled: true
auto_sync_draft: false
auto_sync_ready: true
vetters_override:
- alangou
- derekwaynecarr
- drew
- elezar
- johnnygreco
- johntmyers
- kirit93
- krishicks
- matthewgrossman
- mrunalp
- pimlock
- purp
- SDAChess
- shailendra-nv
- sjenning
- TaylorMutch
- zredlined
1 change: 0 additions & 1 deletion .github/workflows/release-canary.yml
Original file line number Diff line number Diff line change
Expand Up @@ -359,7 +359,6 @@ jobs:
--set server.disableTls=true \
--set server.auth.allowUnauthenticatedUsers=true \
--set "server.telemetryEnabled=${OPENSHELL_TELEMETRY_ENABLED}" \
--set supervisor.sandboxRuntime.networkPolicyEnforced=true \
--wait --timeout 5m

- name: Verify gateway pod is Ready
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/release-tag.yml
Original file line number Diff line number Diff line change
Expand Up @@ -142,7 +142,7 @@ jobs:
with:
candidate_ref: ${{ inputs.tag || github.ref_name }}
fail-on-codex-findings: true
fail-on-static-findings: true
fail-on-static-findings: false
images: |
ghcr.io/nvidia/openshell/gateway:${{ needs.compute-versions.outputs.source_sha }}
ghcr.io/nvidia/openshell/sandbox:${{ needs.compute-versions.outputs.source_sha }}
Expand Down
31 changes: 20 additions & 11 deletions CI.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,12 @@ For local test commands see [TESTING.md](TESTING.md). For PR conventions see [CO

PR CI that runs on NVIDIA self-hosted runners uses NVIDIA's copy-pr-bot. The bot mirrors trusted PR commits to internal `pull-request/<N>` branches in this repository. The gated workflows trigger on pushes to those branches, not on the original PR.

When a PR is not mirrored automatically, only the GitHub users listed in
`.github/copy-pr-bot.yaml` under `vetters_override` can admit its current
revision with `/ok to test <SHA>`. The list is a snapshot of the codeowners
and selected repository maintainers; update it when those people change.
This setting does not change the bot's automatic trust policy for ready PRs.

`Branch Checks` run automatically after copy-pr-bot mirrors the PR. `Required CI Gates` posts PR-head statuses that verify the mirror exists, is current, and ran the expected push-based workflows. E2E suites are opt-in because they are more expensive and publish temporary images.

Merge queue validation is a second integration gate for `main`. After a PR has passed the required PR-head statuses, a maintainer adds it to the merge queue. GitHub creates a temporary merge-group branch that combines the latest `main`, the queued PR, and any earlier queued PRs. The same required `OpenShell / ...` status contexts are then published against the merge-group SHA before GitHub merges it.
Expand Down Expand Up @@ -97,10 +103,13 @@ Actions or call it from another workflow. All applicable children analyze the
candidate snapshot. Cargo Deny uses its existing NVIDIA self-hosted runner and
CI container.

Tagged releases treat CodeQL, Trivy, Zizmor, Cargo Deny, and Codex Security
findings as failures of the currently implemented qualification profile. A
profile failure does not prevent a pre-release candidate's complete artifact
set from being published, but it does prevent stable publication.
Tagged releases treat Cargo Deny and Codex Security findings as failures of the
currently implemented qualification profile. A profile failure does not prevent
a pre-release candidate's complete artifact set from being published, but it
does prevent stable publication. CodeQL, Trivy, and Zizmor findings are
temporarily informational for tagged releases: the existing findings were
reviewed and accepted for v0.1.0 and will be addressed in 0.1.x releases.
Scanner failures still fail qualification.

```shell
gh workflow run security-scan.yml --ref main \
Expand Down Expand Up @@ -313,21 +322,21 @@ Flow:
6. New commits push to the mirror automatically and re-trigger `Branch Checks` plus any labeled E2E jobs in `Branch E2E Checks`.
7. When the PR is ready to merge, use **Add to merge queue** instead of merging directly. The queue validates the final integration state before updating `main`.

### Forked PR
### PR requiring manual admission

Prerequisites:

- DCO sign-off (`git commit -s`) on every commit. Commit signing is not required for forks - copy-pr-bot trusts forks based on maintainer review, not signing.
- A maintainer must vouch you. See the [Vouch System](AGENTS.md#vouch-system).
- DCO sign-off (`git commit -s`) on every commit. Manual admission does not require cryptographic commit signing.
- First-time external contributors must be vouched. See the [Vouch System](AGENTS.md#vouch-system).

Flow:

1. Open the PR. The vouch check confirms you are vouched (otherwise the PR is auto-closed).
2. copy-pr-bot does not mirror forks automatically. A maintainer reviews the diff and comments `/ok to test <SHA>` with your latest commit SHA.
3. After `/ok to test`, copy-pr-bot mirrors to `pull-request/<N>`. From here the flow is identical to internal PRs: `Required CI Gates` verifies the mirror and required push workflows, and maintainers apply the E2E label when the extra suites are needed.
1. Open the PR. The vouch check confirms first-time external contributors are vouched (otherwise their PRs are auto-closed).
2. If copy-pr-bot does not mirror it automatically, a listed vetter reviews the diff and comments `/ok to test <SHA>` with the latest commit SHA. Fork location alone does not determine whether a PR is mirrored automatically.
3. After `/ok to test`, copy-pr-bot mirrors to `pull-request/<N>`. From here the flow is identical to automatically admitted PRs: `Required CI Gates` verifies the mirror and required push workflows, and maintainers apply the E2E label when the extra suites are needed.
4. When the PR is ready to merge, maintainers add it to the merge queue so the queued integration state is tested before it reaches `main`.

Important: every new commit you push requires another `/ok to test <new-SHA>` from a maintainer before push-based CI will run on it. If a label is applied while the mirror is stale, `E2E Label Help` will post a comment explaining what's needed.
Important: if a PR requires manual admission, every new commit needs another `/ok to test <new-SHA>` from a listed vetter before push-based CI will run on it. If a label is applied while the mirror is stale, `E2E Label Help` will post a comment explaining what's needed.

## Merge queue

Expand Down
9 changes: 6 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,12 +41,14 @@ The installer installs the latest stable release by default. See [Prerelease and
**Kubernetes installation:**

> **Experimental** — the Kubernetes deployment path is under active development. Expect rough edges and breaking changes.
> **Required:** Your cluster CNI MUST enforce Kubernetes `NetworkPolicy` for
> ingress and egress in every sandbox namespace. OpenShell creates the policies,
> but Kubernetes does not verify that the CNI applies them.

Deploy the OpenShell gateway into a Kubernetes cluster from the OCI chart published to GHCR:

```bash
helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart \
--set supervisor.sandboxRuntime.networkPolicyEnforced=true
helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart
```

See [`deploy/helm/openshell/README.md`](deploy/helm/openshell/README.md) for available versions, dev tag conventions, and configuration.
Expand Down Expand Up @@ -298,7 +300,8 @@ curl -LsSf https://raw.githubusercontent.com/NVIDIA/OpenShell/main/install.sh |
OPENSHELL_VERSION=pre sh
```

The installer downloads only the artifact for the current platform and rejects expired candidates during discovery. Installed packages retain the candidate's exact version, such as `0.1.0-pre.3`. Prerelease tags do not create entries on the GitHub Releases page.
The installer checks prerelease tags from newest to oldest, selects an unexpired artifact from a successful release run for the current platform, and downloads only that artifact. Installed packages retain the candidate's exact version, such as `0.1.0-pre.3`. Prerelease tags do not create entries on the GitHub Releases page.
On Linux, prereleases and explicit release tags use Debian or RPM packages even if `snap` is installed.

The rolling [`dev` release](https://github.com/NVIDIA/OpenShell/releases/tag/dev) does not require GitHub authentication:

Expand Down
16 changes: 14 additions & 2 deletions architecture/build.md
Original file line number Diff line number Diff line change
Expand Up @@ -324,6 +324,12 @@ positive canary uses system Docker; negative preflight coverage verifies that
the installer rejects both missing Docker and the Docker Snap before installing
OpenShell. Its Debian lane removes snapd before running the installer so Snap
precedence cannot change the package under test.
Explicit release tags and the `pre` alias bypass Snap selection and use the
native Debian or RPM package path even when `snap` is available. The `pre` alias
checks matching Git tags in version order, then looks up the exact platform
artifact and verifies the release run instead of listing every repository
artifact.

Snapd runs the gateway as a root-owned system service. Its generated client
certificates reside in root-owned snap state and are unavailable to ordinary CLI
users, so the Snap uses plaintext loopback transport and enables unauthenticated
Expand Down Expand Up @@ -372,6 +378,10 @@ the release tag.

Required checks run on GitHub Actions. Pull-request workflows that use NVIDIA self-hosted runners trigger from copy-pr-bot mirror branches, so trusted PRs are mirrored into `pull-request/<N>` branches before those workflows run. `main` also uses GitHub merge queue so the final queued integration commit is validated before it merges.

For PRs that need manual admission, copy-pr-bot accepts `/ok to test <SHA>`
only from the explicit `vetters_override` list in `.github/copy-pr-bot.yaml`.
This list is maintained separately from the bot's automatic PR trust policy.

The high-level CI model:

1. PR-context gate jobs publish required statuses for the PR head commit.
Expand Down Expand Up @@ -526,8 +536,10 @@ RFC's complete qualification coverage.
The tagged release workflow calls the aggregate Security Scan after publishing
the candidate's commit-addressed gateway, sandbox, and supervisor images. CodeQL,
Trivy, Cargo Deny, and Actionlint/Zizmor run for every release tag; Codex Security
also runs for pre-release tags. High or Critical findings and scanner failures
fail qualification.
also runs for pre-release tags. Scanner failures, Cargo Deny advisories, and
High or Critical Codex Security findings fail qualification. CodeQL, Trivy, and
Zizmor findings are temporarily informational while the findings accepted for
v0.1.0 are addressed in 0.1.x releases.

The `Release Qualification` job aggregates security, conformance, feature,
Docker E2E, and VM E2E results. The currently implemented profile gates stable
Expand Down
2 changes: 1 addition & 1 deletion architecture/compute-runtimes.md
Original file line number Diff line number Diff line change
Expand Up @@ -301,7 +301,7 @@ delete, reconciliation removes the row; otherwise it can remain `Deleting`.
|---|---|---|---|
| Docker | Local development with Docker available. | Capability-free workload container. | Uses `network_mode=none`; a separate capability-free supervisor container mediates egress and access over a private daemon-local Unix socket volume. |
| Podman | Existing rootless driver. | Container. | Not converted by this isolation stack. |
| Kubernetes | Cluster deployment through Helm. | Capability-free sandbox Pod. | Uses one namespace-wide empty-egress workload NetworkPolicy and a separate capability-free supervisor Pod over mutually authenticated TLS. It requires an enforcing CNI and trusted sandbox namespace. |
| Kubernetes | Cluster deployment through Helm. | Capability-free sandbox Pod. | Always creates a namespace-wide empty-egress workload NetworkPolicy and a separate capability-free supervisor Pod over mutually authenticated TLS. It requires an enforcing CNI and trusted sandbox namespace; the Kubernetes API does not attest policy enforcement. |
| VM | Experimental microVM isolation. | Per-sandbox libkrun or QEMU VM. | The NIC-less guest runs `openshell-sandbox` as PID 1; host `openshell-supervisor` owns gateway networking and reaches the guest over vsock. |
| Extension | Out-of-tree drivers operated alongside the gateway. | Whatever boundary the driver implements. | Selected by a custom `compute_drivers = ["<name>"]` entry with `[openshell.drivers.<name>].socket_path`, or at launch time by pairing `--drivers <name>` with `--compute-driver-socket=<path>`. A launch-time endpoint may use a canonical built-in name to preserve its driver-config key while replacing in-process construction. The gateway connects to an operator-provisioned UDS, snapshots `GetCapabilities`, and dispatches all sandbox lifecycle calls through `compute_driver.proto`. The driver process and socket lifecycle are operator-owned; the gateway does not spawn, supervise, or remove unmanaged extension drivers. The trust boundary is the socket's filesystem permissions: the operator must ensure only the gateway uid can read/write it. |

Expand Down
5 changes: 3 additions & 2 deletions architecture/sandbox-limits.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,8 +72,9 @@ budgets as new activity.
Middleware also validates every non-body envelope component. Important examples
include 64 KiB service config, 4 KiB request context, 32 KiB target data, 128
request headers totaling 64 KiB, 64 header mutations, 32 findings per stage,
and 64 metadata entries. The detailed external contract lives in
[Supervisor Middleware](../docs/extensibility/supervisor-middleware.mdx).
and 64 metadata entries. The external contract lives in
`proto/supervisor_middleware.proto`, with service-author guidance in the
[supported middleware operations](../docs/extensibility/supervisor-middleware/operations.mdx).

The work semaphore bounds aggregate buffered middleware input to approximately
`32 × 4 MiB`, plus bounded envelope and parser overhead. It is a concurrency
Expand Down
5 changes: 3 additions & 2 deletions architecture/sandbox.md
Original file line number Diff line number Diff line change
Expand Up @@ -442,8 +442,9 @@ against body-aware L7 policy before later stages or the upstream can observe
them. Requests, results, chain length, execution time, and diagnostics are
bounded; external free-form diagnostic text is not exposed in responses or
security logs. See
[Supervisor Middleware](../docs/extensibility/supervisor-middleware.mdx) for
configuration and protocol details.
[Supervisor Middleware](../docs/extensibility/supervisor-middleware/index.mdx) for
an introduction, or the [configuration guide](../docs/extensibility/supervisor-middleware/configure.mdx)
for service registration and policy attachment.

Inference providers use the same egress path as other external services. An
attached provider profile contributes endpoint and binary policy. The proxy
Expand Down
6 changes: 3 additions & 3 deletions crates/openshell-driver-kubernetes/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -81,9 +81,9 @@ and permits OpenShell supervisor Pods to reach the sandbox TLS port. The
authenticated Sandbox Protocol binds each connection to the exact sandbox and
supervisor Pod identities. Supervisors have normal egress for gateway, DNS,
and policy-approved upstream connections unless an operator policy restricts
them. Set
`sandbox_runtime.network_policy_enforced = true` only after verifying that the cluster
CNI enforces ingress and egress `NetworkPolicy` for sandbox namespaces.
them. The cluster CNI must enforce ingress and egress `NetworkPolicy` for every
sandbox namespace. Kubernetes accepts policy objects without confirming
enforcement, so operators must verify CNI support before running sandboxes.

Each sandbox generation uses two immutable bootstrap Secrets. A trusted init
container stages the sandbox bootstrap into memory, and the sandbox removes it
Expand Down
23 changes: 0 additions & 23 deletions crates/openshell-driver-kubernetes/src/config.rs
Original file line number Diff line number Diff line change
Expand Up @@ -78,30 +78,20 @@ pub const DEFAULT_WORKSPACE_STORAGE_SIZE: &str = "2Gi";
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(default, deny_unknown_fields)]
pub struct KubernetesSandboxRuntimeConfig {
/// Explicit operator assertion that the cluster CNI enforces
/// `networking.k8s.io/v1` `NetworkPolicy` for the sandbox namespaces.
pub network_policy_enforced: bool,
/// TCP port exposed by the workload boundary to its paired control pod.
pub boundary_port: u16,
}

impl Default for KubernetesSandboxRuntimeConfig {
fn default() -> Self {
Self {
network_policy_enforced: false,
boundary_port: 5500,
}
}
}

impl KubernetesSandboxRuntimeConfig {
pub fn validate(&self) -> Result<(), String> {
if !self.network_policy_enforced {
return Err(
"sandbox_runtime.network_policy_enforced must be true after the operator has verified CNI NetworkPolicy enforcement"
.to_string(),
);
}
if self.boundary_port < 1024 {
return Err("sandbox_runtime.boundary_port must be at least 1024".to_string());
}
Expand Down Expand Up @@ -887,19 +877,6 @@ mod tests {
assert!(cfg.workspace_storage_class.is_empty());
}

#[test]
fn sandbox_runtime_requires_network_policy_enforcement_acknowledgement() {
let mut cfg = KubernetesComputeConfig::default();
assert!(
cfg.validate_proxy_uid()
.unwrap_err()
.contains("network_policy_enforced")
);

cfg.sandbox_runtime.network_policy_enforced = true;
cfg.validate_proxy_uid().unwrap();
}

#[test]
fn serde_rejects_sidecar_binary_identity_field() {
let json = serde_json::json!({
Expand Down
8 changes: 0 additions & 8 deletions crates/openshell-driver-kubernetes/src/main.rs
Original file line number Diff line number Diff line change
Expand Up @@ -128,13 +128,6 @@ struct Args {
#[arg(long, env = "OPENSHELL_SUPERVISOR_IMAGE_PULL_POLICY")]
supervisor_image_pull_policy: Option<KubernetesImagePullPolicy>,

#[arg(
long,
env = "OPENSHELL_K8S_SANDBOX_RUNTIME_NETWORK_POLICY_ENFORCED",
default_value_t = false
)]
sandbox_runtime_network_policy_enforced: bool,

#[arg(
long,
env = "OPENSHELL_K8S_SANDBOX_RUNTIME_BOUNDARY_PORT",
Expand Down Expand Up @@ -270,7 +263,6 @@ async fn main() -> Result<()> {
.unwrap_or_else(openshell_core::config::default_supervisor_image),
supervisor_image_pull_policy: args.supervisor_image_pull_policy,
sandbox_runtime: KubernetesSandboxRuntimeConfig {
network_policy_enforced: args.sandbox_runtime_network_policy_enforced,
boundary_port: args.sandbox_runtime_boundary_port,
},
https_proxy: args.https_proxy,
Expand Down
Loading
Loading