[pull] main from NVIDIA:main

.agents/skills/build-openshell-mxc-windows/SKILL.md

@@ -135,7 +135,7 @@ from this skill.
 | `CARGO_TARGET_DIR` | `target` under repo root | Override Cargo output location. Use a short absolute path when x64-to-ARM64 builds approach Windows path-length limits. |
 | `Z3_LIBRARY_PATH_OVERRIDE` | unset | Directory containing an x64 system `libz3.lib`; not valid for ARM64. |
 | `Z3_SYS_Z3_HEADER` | unset | Full `z3.h` path required with a system Z3 library. |
-| `Z3_SYS_Z3_VERSION` | `4.16.0` | Pinned official prebuilt Z3 release selected by the wrapper. |
+| `Z3_SYS_Z3_VERSION` | `5.1.0` | Pinned official prebuilt Z3 release selected by the wrapper. |
 | `READ_ONLY_GITHUB_TOKEN` | unset | Optional token for the Z3 release lookup; GitHub Actions supplies `github.token`. |
 | `RUSTC_WRAPPER` | inherited | The wrapper resolves an available command to an absolute path. If it is unavailable, the wrapper warns and continues without compiler caching. |
 

.agents/skills/build-openshell-mxc-windows/reference.md

@@ -69,7 +69,7 @@ For ARM64, verify the Visual Studio instance contains the ARM64 MSVC tools,
 ARM64 Spectre-mitigated libraries, Clang tools, CMake tools, and a Windows SDK.
 Clang supplies host-native `libclang.dll` for `bindgen` and `clang-cl.exe` for
 ARM64 crypto dependencies such as `aws-lc-sys`. Native and
-x64-to-ARM64 builds use the official prebuilt Z3 4.16.0 static library for the
+x64-to-ARM64 builds use the official prebuilt Z3 5.1.0 static library for the
 target architecture. An x64-to-ARM64 check/build discovers and adds host-native
 Ninja to `PATH`, while the crypto crates select `clang-cl`. Use a short
 `CARGO_TARGET_DIR` if Windows path-length limits are reached.

.github/workflows/release-tag.yml

@@ -141,7 +141,11 @@ jobs:
     uses: ./.github/workflows/security-scan.yml
     with:
       candidate_ref: ${{ inputs.tag || github.ref_name }}
-      allow-high-critical: false
+      # Temporary observation mode for the pre-existing CodeQL, Trivy, and
+      # Zizmor backlog. Remove after baseline/delta enforcement is implemented.
+      # Cargo Deny, Codex Security, and scanner execution errors remain fatal.
+      fail-on-codex-findings: true
+      fail-on-static-findings: false
       images: |
         ghcr.io/nvidia/openshell/gateway:${{ needs.compute-versions.outputs.source_sha }}
         ghcr.io/nvidia/openshell/sandbox:${{ needs.compute-versions.outputs.source_sha }}

.github/workflows/security-scan.yml

@@ -18,9 +18,13 @@ on:
         description: Allow a full Codex scan when no previous stable tag exists
         default: false
         type: boolean
-      allow-high-critical:
-        description: Report HIGH/CRITICAL findings without failing; scanner errors still fail
-        default: false
+      fail-on-codex-findings:
+        description: Fail when Codex Security reports findings
+        default: true
+        type: boolean
+      fail-on-static-findings:
+        description: Fail when CodeQL, Trivy, or Zizmor reports HIGH/CRITICAL findings
+        default: true
         type: boolean
       images:
         description: Newline-separated image references for Trivy
@@ -44,9 +48,13 @@ on:
         description: Allow a full Codex scan when no previous stable tag exists
         default: false
         type: boolean
-      allow-high-critical:
-        description: Report HIGH/CRITICAL findings without failing; scanner errors still fail
-        default: false
+      fail-on-codex-findings:
+        description: Fail when Codex Security reports findings
+        default: true
+        type: boolean
+      fail-on-static-findings:
+        description: Fail when CodeQL, Trivy, or Zizmor reports HIGH/CRITICAL findings
+        default: true
         type: boolean
       images:
         description: Newline-separated image references for Trivy
@@ -79,7 +87,7 @@ jobs:
       candidate_ref: ${{ inputs.candidate_ref }}
       stable_ref: ${{ inputs.stable_ref }}
       allow_full_bootstrap: ${{ inputs.allow_full_bootstrap }}
-      fail-on-findings: ${{ !inputs.allow-high-critical }}
+      fail-on-findings: ${{ inputs.fail-on-codex-findings }}
       upload_sarif: true
     secrets:
       CODEX_SECURITY_API_KEY: ${{ secrets.CODEX_SECURITY_API_KEY }}
@@ -92,7 +100,7 @@ jobs:
       security-events: write
     with:
       candidate_ref: ${{ inputs.candidate_ref }}
-      fail-on-findings: ${{ !inputs.allow-high-critical }}
+      fail-on-findings: ${{ inputs.fail-on-static-findings }}
 
   trivy:
     name: Trivy
@@ -103,7 +111,7 @@ jobs:
       security-events: write
     with:
       candidate_ref: ${{ inputs.candidate_ref }}
-      fail-on-findings: ${{ !inputs.allow-high-critical }}
+      fail-on-findings: ${{ inputs.fail-on-static-findings }}
       severity: HIGH,CRITICAL
       ignore-unfixed: false
       images: ${{ inputs.images }}
@@ -129,6 +137,6 @@ jobs:
       security-events: write
     with:
       candidate_ref: ${{ inputs.candidate_ref }}
-      fail-on-findings: ${{ !inputs.allow-high-critical }}
+      fail-on-findings: ${{ inputs.fail-on-static-findings }}
     secrets:
       CACHIX_AUTH_TOKEN: ${{ secrets.CACHIX_AUTH_TOKEN }}

CI.md

@@ -97,10 +97,18 @@ Actions or call it from another workflow. All applicable children analyze the
 candidate snapshot. Cargo Deny uses its existing NVIDIA self-hosted runner and
 CI container.
 
+Tagged releases temporarily run CodeQL, Trivy, and Zizmor findings in
+observation mode while the existing backlog is triaged. Cargo Deny advisories,
+Codex Security findings, and scanner execution errors still block publication.
+The temporary `fail-on-static-findings: false` override must be removed when
+baseline/delta enforcement is implemented; it does not waive Cargo Deny, Codex
+Security, or scanner execution failures.
+
 ```shell
 gh workflow run security-scan.yml --ref main \
   -f candidate_ref=v0.1.1-pre.1 \
-  -F allow-high-critical=true
+  -F fail-on-codex-findings=false \
+  -F fail-on-static-findings=false
 ```
 
 To integrate it into a larger workflow, run it after the job that pushes the
@@ -125,7 +133,8 @@ jobs:
         ${{ needs.build.outputs.gateway_image }}
         ${{ needs.build.outputs.sandbox_image }}
       charts: ${{ needs.build.outputs.chart_ref }}
-      allow-high-critical: false
+      fail-on-codex-findings: true
+      fail-on-static-findings: true
     secrets:
       CODEX_SECURITY_API_KEY: ${{ secrets.CODEX_SECURITY_API_KEY }}
       CACHIX_AUTH_TOKEN: ${{ secrets.CACHIX_AUTH_TOKEN }}
@@ -140,20 +149,22 @@ publishes SARIF, including Codex results for manual parent runs. Codex keeps its
 release-train category on `main`; CodeQL, Trivy, and workflow reports publish
 against the candidate tag and commit. Existing standalone triggers stay active.
 
-The parent fails on HIGH/CRITICAL findings from Codex, CodeQL, Trivy, and Zizmor.
-Set `allow-high-critical: true` to report those findings without failing; it
-defaults to `false`. Scanner setup, execution, and report publication errors
-still fail. Reports are published before the finding threshold is enforced.
+The parent fails on Codex findings when `fail-on-codex-findings` is `true` and
+on HIGH/CRITICAL findings from CodeQL, Trivy, and Zizmor when
+`fail-on-static-findings` is `true`. Both inputs default to `true`. Set either
+input to `false` to make only that finding class informational. Scanner setup,
+execution, and report publication errors still fail. Reports are published
+before the finding threshold is enforced.
 
 Codex uses each finding's severity; CodeQL uses the rule's security score
 (at least 7.0); Trivy uses `HIGH,CRITICAL`, including vulnerabilities without an
 upstream fix; Zizmor uses its High severity. Actionlint remains informational.
 Existing scanner exceptions still apply.
 
 Cargo Deny runs only `cargo deny check advisories` in the parent. It keeps its
-native failure behavior and configured exceptions, regardless of
-`allow-high-critical`; it has no HIGH/CRITICAL filter. Its standalone runs still
-check all dependency policies.
+native failure behavior and configured exceptions regardless of either finding
+threshold input; it has no HIGH/CRITICAL filter. Its standalone runs still check
+all dependency policies.
 
 Codex scans the cumulative diff from `stable_ref` to `candidate_ref`. Both inputs
 are tag names, not arbitrary commit SHAs. Omit `stable_ref` to resolve the previous
@@ -377,7 +388,7 @@ These workflows run after merge to publish dev/tagged artifacts and verify them.
 | File | Role |
 |---|---|
 | `.github/workflows/release-dev.yml` | Publishes the rolling `dev` build on every push to `main`. Builds gateway, sandbox, and supervisor images and binaries, packages, wheels, and pushes the Helm chart as `oci://ghcr.io/nvidia/openshell/helm-chart:0.0.0-dev` (plus an immutable `0.0.0-dev.<sha>` pin). Also dispatchable manually. |
-| `.github/workflows/release-tag.yml` | Publishes tagged stable releases and manually dispatched pre-releases. Its automatic tag trigger excludes `-pre.*`. Security Scan gates release publication. |
+| `.github/workflows/release-tag.yml` | Publishes tagged stable releases and manually dispatched pre-releases. Its automatic tag trigger excludes `-pre.*`. Security Scan gates release publication on Cargo Deny advisories, Codex Security findings, and scanner execution errors while the static-analysis backlog remains in temporary observation mode. |
 | `.github/workflows/release-canary.yml` | Smoke-tests published dev artifacts on `macos`, `ubuntu`, `fedora`, and `kubernetes` (kind + Helm) runners. Each job reaches its gateway and creates, exercises, and deletes a sandbox. It runs automatically after `Release Dev` succeeds and supports manual dispatch (`gh workflow run release-canary.yml --ref <branch>`). See the `test-release-canary` skill for the playbook and local kind reproduction. |
 
 ## Required status contexts

CONTRIBUTING.md

@@ -343,7 +343,7 @@ cargo build -p openshell-prover-cli --features bundled-z3
 For x86-64 and ARM64 Windows MSVC builds, use one of these Z3 paths:
 
 - Prebuilt Z3 (the default for `windows:*` tasks): `z3-sys` downloads the
-  pinned Z3 4.16.0 GitHub release for the target architecture on the first
+  pinned Z3 5.1.0 GitHub release for the target architecture on the first
   build. Cargo reuses the extracted archive from its target directory. Windows
   CI authenticates the GitHub API request with `READ_ONLY_GITHUB_TOKEN` and
   preserves the archive in the architecture-specific Cargo target cache. For
@@ -381,8 +381,8 @@ To use a local x64 Z3 release instead of the prebuilt download, set
 `Z3_LIBRARY_PATH_OVERRIDE` and `Z3_SYS_Z3_HEADER` before running the task:
 
 ```powershell
-$env:Z3_LIBRARY_PATH_OVERRIDE='C:\path\to\z3-4.16.0-x64-win\bin'
-$env:Z3_SYS_Z3_HEADER='C:\path\to\z3-4.16.0-x64-win\include\z3.h'
+$env:Z3_LIBRARY_PATH_OVERRIDE='C:\path\to\z3-5.1.0-x64-win\bin'
+$env:Z3_SYS_Z3_HEADER='C:\path\to\z3-5.1.0-x64-win\include\z3.h'
 mise run --skip-tools windows:build:x64
 ```
 

Cargo.toml

@@ -144,7 +144,7 @@ uuid = { version = "1.10", features = ["v4"] }
 signal-hook = "0.3"
 
 # SMT solver (uses system libz3; enable z3/bundled via the prover's bundled-z3 feature for local dev without system z3)
-z3 = "0.20"
+z3 = "0.21"
 
 [workspace.lints.rust]
 unsafe_code = "warn"

architecture/build.md

@@ -512,8 +512,12 @@ The tagged release workflow calls the aggregate Security Scan after publishing
 the candidate's commit-addressed gateway, sandbox, and supervisor images. CodeQL,
 Trivy, Cargo Deny, and Actionlint/Zizmor run for every release tag; Codex Security
 also runs for pre-release tags. The release job depends on the aggregate result,
-which fails on scanner errors, Cargo Deny advisories, and unexcepted High or
-Critical findings.
+which fails on scanner errors, Cargo Deny advisories, and Codex Security
+findings. CodeQL, Trivy, and Zizmor findings temporarily remain informational
+while their existing backlog is triaged. Remove the release workflow's
+`fail-on-static-findings: false` override when baseline/delta enforcement is
+implemented so new High or Critical findings become blocking without requiring
+the historical backlog to reach zero first.
 
 `release-auto-tag.yml` runs at 14:00 Europe/Zurich on weekdays (including daylight
 saving time changes) and supports manual dispatch. Maintainers start weekday

architecture/gateway.md

@@ -440,6 +440,82 @@ Domain objects use shared metadata: stable server-generated IDs, human-readable
 names, creation timestamps, and labels. Crate-level details live in
 `crates/openshell-core/README.md`.
 
+### Watch streams
+
+`WatchSandbox` merges three per-sandbox sources into one client stream: status
+snapshots, server/sandbox logs, and platform events. Logs and platform events
+are resumable; a shared per-sandbox allocator stamps each with a `cursor`.
+Cursor-ordered delivery is guaranteed for the replay phase: on resume the
+buffered events from both sources are sorted before emission. Live events are
+monotonic within each source, but the two sources are read independently, so a
+client should order across sources by `cursor` rather than by arrival. Status
+snapshots and warnings are re-read on demand and carry an empty cursor.
+
+#### Cursor spaces
+
+A sandbox's cursors live in a **cursor space**: a `{epoch, seq}` pair, where the
+epoch is a UUID minted on the first publish and `seq` counts from 1. The epoch
+is dropped by `TracingLogBus::remove`, so a teardown — or a gateway restart —
+retires the space, and the next publish mints a new one. A sequence number alone
+cannot distinguish a caught-up client from one holding a cursor out of a space
+that no longer exists, because the replacement space reuses the same numbers;
+the epoch answers *which counter issued this*, which is the question resume
+validation actually has to ask. Behind multiple replicas the same rule makes a
+reconnect to a different replica fail loudly rather than return the wrong
+events.
+
+On the wire a cursor is an opaque, fixed-width token. Clients may only compare
+two cursors from one stream and keep the greater; the encoding zero-pads `seq`
+so that byte-wise comparison matches sequence order, which is what lets every
+SDK track a high-water mark without parsing. A stream only ever observes one
+epoch — a reset closes both resumable broadcast receivers, ending the stream
+rather than switching spaces mid-flight — so that comparison is always well
+defined where clients are allowed to use it. The gateway does not rely on it:
+server-side ordering runs on the raw `u64` seq carried alongside each event in
+`CursoredEvent`, never on the token.
+
+The gateway holds a bounded in-memory tail per sandbox. Loss is reported with
+two distinct, documented behaviors:
+
+- **Recoverable lag** — a broadcast receiver falls behind and the server skips
+  ahead. The stream emits a `SandboxStreamWarning` event and continues. Since
+  cursors are opaque, the warning is the client's only signal.
+- **Unrecoverable gap** — the server sends a snapshot, then terminates with
+  `OUT_OF_RANGE`. Three cases reach it: the tail was trimmed past the requested
+  cursor, the cursor's epoch does not match the sandbox's current space, or no
+  space exists because nothing has been published since teardown. The status
+  tells the client to restart with an empty cursor; retrying the same token
+  fails identically. A token the gateway could not have issued is rejected
+  earlier, as `INVALID_ARGUMENT` on the call itself.
+
+Both resumable sources draw from one cursor space, so the server merges them by
+seq before emitting rather than draining each in turn: on resume it replays only
+events after the client's cursor, and without one it replays each bus's retained
+tail. Either way the batch leaves in ascending cursor order. The two tails are
+bounded independently (`log_tail_lines` and `event_tail`), so merging orders
+whatever each bus kept; it does not align their depths.
+
+The broadcast receivers are subscribed before replay, so an event buffered during
+initialization could appear in both replay and the live receiver; the producer
+tracks the highest replayed seq and suppresses live events at or below it, so
+each event is delivered once. That mark is per source. The two tails are read at
+different instants and bounded independently, so one shared mark would let the
+deeper source censor the shallower one — with `event_tail` unset the mark rises
+to the newest buffered log while no platform event is replayed at all, and
+platform events published during initialization are discarded as duplicates of a
+replay that never ran. Subscribing never mints a cursor space, so a
+resume against a torn-down sandbox cannot create the space its stale cursor is
+then checked against. Clients track the highest observed `cursor` and pass it as
+`resume_after_cursor` on reconnect.
+
+The epoch is validated twice on resume: once before reading the tails and again
+once both are in hand, before anything is emitted. The check and each read take
+their locks separately, so a teardown plus a republish can retire the validated
+space and install a replacement in between; the reads would then apply the old
+space's seq to the replacement's buffers, and a trimmed-range check that only
+compares numbers would report no gap while skipping the replacement's lower
+events. The second look ends the stream with `OUT_OF_RANGE` instead.
+
 ## Persistence
 
 The gateway persistence layer is a protobuf object store. Domain services store
@@ -959,6 +1035,26 @@ alone — instead an unanswered keepalive on a wedged or orphaned relay closes t
 channel and returns the exec with an error. Once a command reports its exit
 status, the gateway also bounds how long it waits for the trailing channel close.
 
+Interactive exec treats normal request-stream EOF as the end of stdin and resize
+input. The gateway sends SSH EOF while keeping the output channel open until
+command completion. Input errors terminate the operation rather than masquerading
+as normal EOF. The input and output pumps are owned by the exec operation, so
+timeout or response abandonment cannot leave a detached stdin task behind.
+The pumps share polling fairly, and request processing yields cooperatively even
+for ignored resize messages, so sustained input cannot monopolize the operation.
+
+Go and TypeScript interactive-exec helpers distinguish process exit from stream
+completion. They consume the final gRPC status before reporting success and retain
+an observed exit code if transport completion fails. Callers must drain output
+concurrently with waiting for completion.
+
+TypeScript starts interactive exec eagerly and uses a bounded output queue between
+the background receiver and the consumer. Cancellation wakes a receiver blocked
+on that queue. Go exposes input closure through an optional session capability,
+preserving the original interface for existing implementations. TypeScript also
+preserves its original session interface; SDK-created sessions expose lifecycle
+controls through an extended interface.
+
 `ForwardTcp` is the client-facing byte stream for SSH and service forwarding.
 The first frame is a `TcpForwardInit` that carries the workspace-scoped sandbox
 name, an authorization token from `CreateSshSession`, and an explicit target: