Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions .agents/skills/helm-dev-environment/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,14 @@ mise run helm:skaffold:dev
mise run helm:skaffold:run
```

Resource admission defaults to enabled and caller driver config to disabled.
Driver-config scenarios need an explicit `allowDriverConfig` opt-in; external
attachments also need administrator-controlled approval labels in the target
namespace. GPU attachments and operator-selected image-pull Secrets are exempt
from labels. Managed workspace image-pull Secrets are copied from the configured
source in the gateway namespace; do not grant approval to the gateway database
PVC or disable admission to make tests pass.

The Skaffold flow builds distinct `gateway`, `sandbox`, and `supervisor` images
and deploys the OpenShell Helm chart. The Kubernetes driver creates a
capability-free workload Pod and a directly managed capability-free supervisor
Expand Down
31 changes: 16 additions & 15 deletions .github/workflows/branch-e2e.yml
Original file line number Diff line number Diff line change
Expand Up @@ -247,21 +247,21 @@ jobs:
]

# Run driver-specific integration tests:
# TODO: This should be added as soon as we have driver-specific tests enabled in tmachine.
# driver-specific:
# needs: prepare-integration
# permissions:
# actions: read
# contents: read
# packages: read
# uses: ./.github/workflows/integration-runner.yml
# with:
# category: driver-specific
# source-sha: ${{ needs.prepare-integration.outputs.source_sha }}
# integration-inputs-artifact-id: ${{ needs.prepare-integration.outputs.integration_inputs_artifact_id }}
# test-matrix: >-
# [
# ]
driver-specific-integration:
needs: prepare-integration
permissions:
actions: read
contents: read
packages: read
uses: ./.github/workflows/integration-runner.yml
with:
category: driver-specific
source-sha: ${{ needs.prepare-integration.outputs.source_sha }}
integration-inputs-artifact-id: ${{ needs.prepare-integration.outputs.integration_inputs_artifact_id }}
test-matrix: >-
[
{"environment":"fedora-podman-rootless","installer":"binaries","testsuite":"driver-podman"}
]

docker-e2e:
needs: [pr_metadata, build-binaries, build-images]
Expand Down Expand Up @@ -451,6 +451,7 @@ jobs:
- pr_metadata
- conformance-integration
- feature-specific-integration
- driver-specific-integration
- docker-e2e
- vm-e2e
- docker-external-driver-e2e
Expand Down
1 change: 1 addition & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

29 changes: 24 additions & 5 deletions architecture/compute-runtimes.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,22 @@ the common protocol owns process, identity, TCP, DNS, and forwarding semantics.

## Driver Contract

External resource admission is an operator-owned boundary shared by drivers.
The gateway gates caller driver JSON independently from attachment approval.
Drivers resolve the complete effective attachment inventory against authoritative
resource labels before launch and on reuse. Missing labels or an unsupported
resolver deny access; GPU attachments are an explicit temporary exception.
Fresh sandbox-private resources instead require verified provisioning ownership.
Workload metadata must not grant approval or override admission evidence.

The shared evaluator lives in `openshell-core`; native resolution remains in
each driver. External drivers acknowledge the effective versioned policy through
capabilities, and policy mismatch prevents activation or new launch operations.
Trusted deployment configuration can explicitly disable label admission, but
that opt-out does not waive other ownership and isolation checks. This boundary
assumes operators control approval metadata and runtime resource replacement;
it does not provide atomic mount authorization or instantaneous revocation.

Each runtime receives a sandbox spec and canonical policy from the gateway and
is responsible for:

Expand Down Expand Up @@ -449,11 +465,14 @@ management. RBAC uses a namespace-scoped Role.
Each new namespace receives a ServiceAccount and the configured gateway-only
SSH ingress NetworkPolicy. Configured image-pull Secrets are copied from the
driver's source namespace on every sandbox create so registry credential
rotations propagate. The namespace also copies OpenShift SCC UID-range and
supplemental-group annotations from the gateway namespace when present. The
driver deletes the namespace during workspace deletion. The workspace remains
durably `Terminating` until the Kubernetes API accepts namespace cleanup, so a
transient failure can be retried. Namespace deletion uses the fetched UID as a
rotations propagate. Their names are operator-selected gateway configuration,
not caller attachments. Copies carry gateway and workspace ownership labels; an
unrelated existing target is never adopted. The namespace also copies
OpenShift SCC UID-range and supplemental-group annotations from the gateway
namespace when present. The driver deletes the namespace during workspace
deletion. The workspace remains durably `Terminating` until the Kubernetes API
accepts namespace cleanup, so a transient failure can be retried. Namespace
deletion uses the fetched UID as a
precondition to avoid deleting a replacement namespace. Requires a non-empty
`gateway_id` (validated as a
DNS-1123 label at startup) so the namespace prefix fits within the K8s 63-character
Expand Down
29 changes: 28 additions & 1 deletion architecture/gateway.md
Original file line number Diff line number Diff line change
Expand Up @@ -95,7 +95,25 @@ in-memory extension. Replay reauthorizes original and current effective scopes,
requires the same effective payload, and reruns current interceptor validation.
Interceptors cannot mutate the request UUID. Server-marked replay suppresses
post-commit observation, which remains best-effort rather than an outbox.
Credential capabilities and streaming execution require separate contracts.
Credential capabilities require separate contracts.

Both exec RPCs share keyed admission but defer completion to the SSH producer.
The initial interactive Start uses the exec request schema under a distinct RPC
namespace; later stdin and resize frames are not replayable inputs. Admission
resolves the public sandbox name and workspace, durably binds both original and
effective sandbox UUIDs, and rejects same-name replacements. When the original
and effective selectors match, both authorization lookups must resolve the same
workspace and sandbox identities before admission. A different effective target
is allowed only when the selector changes. The owner checks the effective UUID
again before relay opening, then hands its CAS-only finalizer to the owned producer,
releasing the shared admission-worker permit after handoff. Only a confirmed
remote exit records a terminal marker and starts 24-hour retention. Synthetic
timeouts, disconnects without exit confirmation, and persistence failures leave
permanent unresolved claims. Duplicates never launch, attach, or replay output:
pending records report uncertainty, and terminal records report stream
unavailability. Existing transport cancellation behavior remains unchanged.
Exec timeouts retain the public duration's precision and presence: an absent
timeout is unbounded, while an explicit zero is a finite timeout.

The gateway listens on one service port and multiplexes gRPC and HTTP traffic.
The default local single-user deployment mode is mTLS user authentication:
Expand Down Expand Up @@ -1026,6 +1044,15 @@ The same relay pattern backs interactive SSH, command execution, file sync, and
local service forwarding. The gateway tracks live sessions in memory and
persists session records so tokens can expire or be revoked.

Graceful gateway shutdown closes supervisor-session admission before stopping
local compute. It then signals the remaining control sessions to exit and waits
up to ten seconds for their cleanup, including conditional deletion of persisted
ownership. Pending connection setup and sessions already removed from the live
registry remain tracked until cleanup finishes. This lets a replacement
supervisor claim ownership immediately after restart without deleting a newer
replica's claim. An incomplete drain is reported as a shutdown error. Closing
these control sessions does not stop Kubernetes-owned workloads.

Relay liveness has two backstops so a reset supervisor session cannot leave a
request parked forever. The gateway runs server-side HTTP/2 keepalive on
supervisor connections, and each exec relay's SSH client uses SSH keepalive: an
Expand Down
1 change: 1 addition & 0 deletions architecture/sandbox-limits.md
Original file line number Diff line number Diff line change
Expand Up @@ -95,6 +95,7 @@ streaming HTTP middleware to use the same process-wide budget.
|---|---:|---|
| Initial CONNECT request headers | 8 KiB | Reject the proxy request. |
| Inspected HTTP/1 request headers | 16 KiB | Reject the request. |
| Streamed HTTP/1 chunk framing | 16 KiB per chunk-size line; 16 KiB and 128 fields for the complete trailer block | End the relay. Chunk payloads pass through a fixed 8 KiB buffer and do not accumulate to the declared chunk size. |
| Credential-rewritten HTTP body | 256 KiB | Reject when rewriting requires a larger buffered body. |
| SigV4 body signing | 10 MiB | Reject when signing requires a larger buffered body. |
| GraphQL request body | 64 KiB default | Policy can set a positive `graphql_max_body_bytes`; there is no shared platform ceiling yet. |
Expand Down
28 changes: 25 additions & 3 deletions architecture/sandbox.md
Original file line number Diff line number Diff line change
Expand Up @@ -205,9 +205,31 @@ The sandbox reserves `SIGUSR2` with a non-restarting no-op handler for these
broker threads; startup rejects a conflicting handler. This signal disposition
is process-global kernel state, while registrations and cancellation state are
owned by the broker. Workload exec resets the caught handler to its default.
This sandbox runtime requires Linux 6.2 or newer for Landlock ABI v3 and treats
`SECCOMP_FILTER_FLAG_WAIT_KILLABLE_RECV` as mandatory so cancelled
notifications cannot race task-memory writes.
This sandbox runtime requires Landlock ABI v3 (Linux 6.2, or an equivalent
vendor backport). The seccomp listener is installed in one of two cancellation
modes, and the launch confirmation enforces the invariant
`cancellation || task_memory_writes_disabled`:

- **Killable** (`SECCOMP_FILTER_FLAG_WAIT_KILLABLE_RECV`, Linux 5.19+): the
notified workload thread waits kill-only, so a non-fatal signal cannot resume
a mediated syscall between notification validation and the broker's result
write. Full mediation, including task-memory output writes.
- **LegacyReadOnly** (kernels < 5.19, e.g. RHEL 9.x / 5.14): the flag is
unavailable (`EINVAL`), so the listener falls back to a plain notifier and the
broker refuses every task-memory *output* write to stay cancellation-safe.
Concretely, in this mode `getpeername`, `accept`/`accept4` **with a non-null
peer-address argument**, and `sendmmsg` paths that write per-message lengths
fail closed with `EOPNOTSUPP`. `accept` with a null address, and socket
creation, `connect`, `bind`, `listen`, `sendto`, and `sendmsg` continue to
work — they use copied inputs, scalar responses, or atomic `ADDFD_SEND`, none
of which write into workload memory. Some server workloads whose accept
wrappers request the peer address will therefore not run until the kernel
provides `WAIT_KILLABLE_RECV` (a distribution backport); outbound-oriented
workloads are unaffected.

Input mediation, DNS/TCP authorization, and outer-fence enforcement are
identical in both modes. The selected mode is emitted in the sandbox
qualification output (`seccomp_listener_mode`).

DNS uses an exact sandbox-local resolver at `127.0.0.53:53`. The driver sets the
nameserver and permits an unprivileged bind to port 53. UDP and TCP DNS requests
Expand Down
8 changes: 5 additions & 3 deletions crates/openshell-cli/src/main.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1465,8 +1465,9 @@ enum SandboxCommands {
#[arg(long)]
memory: Option<String>,

/// Experimental driver-keyed JSON object for driver-specific sandbox settings.
/// Validation behavior is not yet finalized.
/// Driver-keyed JSON object for driver-specific sandbox settings.
/// Disabled unless the gateway administrator enables `allow_driver_config`.
/// External resource attachments still require approval labels.
///
/// For Kubernetes, pass a value such as
/// `{"kubernetes":{"pod":{"node_selector":{"pool":"gpu"}}}}`.
Expand Down Expand Up @@ -1885,7 +1886,8 @@ enum SandboxTemplateCommands {
#[arg(long, num_args = 0..=1, value_name = "COUNT", default_missing_value = "", value_parser = parse_gpu_request)]
gpu: Option<GpuCliRequest>,

/// Experimental driver-keyed JSON object for driver-specific sandbox settings.
/// Driver-keyed JSON object for driver-specific sandbox settings.
/// Requires administrator opt-in; resource admission still applies.
#[arg(long, value_name = "JSON")]
driver_config_json: Option<String>,

Expand Down
2 changes: 2 additions & 0 deletions crates/openshell-cli/src/run.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1897,6 +1897,7 @@ pub async fn sandbox_exec_grpc(
// Make the streaming gRPC call.
let mut stream = client
.exec_sandbox(ExecSandboxRequest {
request_id: String::new(),
sandbox: name.to_string(),
workspace_scope: Some(openshell_core::proto::workspace_selector(
workspace.to_string(),
Expand Down Expand Up @@ -2300,6 +2301,7 @@ async fn sandbox_exec_interactive_grpc(
input_tx
.send(ExecSandboxInput {
payload: Some(exec_sandbox_input::Payload::Start(ExecSandboxRequest {
request_id: String::new(),
sandbox: sandbox.object_name().to_string(),
workspace_scope: Some(openshell_core::proto::workspace_selector(
(sandbox.object_workspace()).to_string(),
Expand Down
1 change: 1 addition & 0 deletions crates/openshell-core/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,7 @@ pub mod proposals;
pub mod proto;
pub mod proto_struct;
pub mod provider_credentials;
pub mod resource_admission;
pub mod rpc_error;
pub mod sandbox_env;
pub mod sandbox_generation;
Expand Down
Loading
Loading