Skip to content

Commit 4f082a5

Browse files
committed
feat(supervisor): apply streamed configuration snapshots
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
1 parent e7729ec commit 4f082a5

15 files changed

Lines changed: 1900 additions & 209 deletions

File tree

‎architecture/gateway.md‎

Lines changed: 9 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -574,8 +574,10 @@ still referenced by a sandbox.
574574

575575
Policy and runtime settings are delivered together through the effective sandbox
576576
config path. A gateway-global policy can override sandbox-scoped policy. The
577-
sandbox supervisor polls for config revisions and hot-reloads dynamic policy
578-
when the policy engine accepts the update.
577+
gateway pushes complete snapshots to active supervisor sessions and periodically
578+
rebuilds them to repair missed delivery. Supervisors hot-reload accepted policy
579+
and acknowledge the exact revision. The legacy poller remains as a mixed-version
580+
compatibility path during this stage.
579581

580582
External supervisor middleware registration is operator-owned configuration
581583
under `[[openshell.supervisor.middleware]]`. At startup the gateway connects to
@@ -640,10 +642,12 @@ process-local supervisor registry. A future HA implementation can resolve the
640642
gateway that owns a session and forward the same typed message without changing
641643
mutation handlers.
642644

643-
Polling remains authoritative during the first rollout stage. Snapshot build,
645+
Current supervisors apply stream snapshots directly. Previous-revision
646+
supervisors retain polling as a rollout fallback, and owner reconciliation
647+
repairs missed or failed delivery from current database state. Snapshot build,
644648
fanout, or enqueue failure cannot fail a mutation that already committed.
645-
Provider snapshots may contain credentials and must not be
646-
persisted or included in logs.
649+
Provider snapshots may contain credentials and must not be persisted or
650+
included in logs.
647651

648652
See [sandbox configuration delivery](sandbox.md#supervisor-configuration-delivery)
649653
for bootstrap, revision, and supervisor application semantics.

‎architecture/sandbox.md‎

Lines changed: 35 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -475,15 +475,18 @@ quickly.
475475

476476
## Supervisor Configuration Delivery
477477

478-
The gateway and supervisor must implement the same internal supervisor protocol
479-
revision. Peers built before the handshake existed report revision zero and are
480-
accepted for one release with a warning and a counter, because sandboxes keep
481-
their supervisor binary until they are recreated. The gateway includes a configuration bootstrap when it accepts a
482-
`ConnectSupervisor` session and can send complete component replacements on the
483-
same stream after policy, settings, or provider state changes.
484-
While polling remains authoritative, optional bootstrap construction has a
485-
one-second budget. The gateway accepts the session without a bootstrap when
486-
that budget expires, so slow credential backends do not block relay reconnects.
478+
The current gateway and supervisor use internal supervisor protocol revision 2.
479+
The gateway accepts Stage 1 revision 1 supervisors through the polling
480+
compatibility path. Peers built before the handshake report revision zero and
481+
remain accepted for one release with a warning and counter because sandboxes
482+
keep their supervisor binary until they are recreated. The gateway includes a
483+
configuration bootstrap when it accepts a `ConnectSupervisor` session and can
484+
send complete component replacements on the same stream after policy, settings,
485+
or provider state changes.
486+
Revision 2 supervisors require a complete bootstrap. The gateway uses the same
487+
bounded 45-second construction window as other snapshot builds and rejects the
488+
connection when construction fails. Revision 1 compatibility sessions retain
489+
the optional one-second bootstrap budget and use polling when it expires.
487490
These payloads describe the latest effective state rather than
488491
the mutation that produced it. The gateway assigns ordering sequences within
489492
each session and component, while each snapshot retains its own content
@@ -503,21 +506,34 @@ without changing publishers. Provider payloads can contain
503506
credentials, so the gateway does not persist or render complete stream
504507
messages in logs.
505508

506-
The supervisor currently parses and ignores stream-delivered configuration.
507-
Polling remains the only path that changes runtime state and repairs dropped or
508-
unavailable delivery. The gateway serializes construction per sandbox and
509-
component, and coalesces repeated mutations into the latest full snapshot. An
510-
enqueue result means only that the local stream queue accepted the message. A
511-
bounded scope fanout scheduler coalesces repeated workspace and global changes,
509+
The supervisor applies stream-delivered configuration through the same runtime
510+
primitives used by the compatibility poller. It reports the requested and
511+
active revisions plus a component-specific outcome on `ConnectSupervisor`.
512+
Sandbox-scoped policy results update only the matching policy-history row, so a
513+
late result cannot mark a newer revision loaded. Explicit local policy remains
514+
authoritative and produces a retained-local-override result.
515+
516+
The gateway keeps one update in flight per session and component. It replaces
517+
the pending snapshot when newer desired state arrives, validates the update ID,
518+
component sequence, component, and requested revision on acknowledgement, then
519+
sends the newest pending snapshot. A 30-second owner reconciliation pass
520+
rebuilds current snapshots for active local sessions. Applied equality
521+
revisions suppress unchanged delivery, while failed or timed-out delivery is
522+
retried from current database state. Reconnect discards session delivery state
523+
and starts with a fresh bootstrap.
524+
525+
Polling remains available during the mixed-version rollout. The gateway
526+
serializes construction per sandbox and component, and coalesces repeated
527+
mutations into the latest full snapshot. An enqueue result means only that the
528+
local stream queue accepted the message. A bounded scope fanout scheduler
529+
coalesces repeated workspace and global changes,
512530
and semaphores sized from the database pool bound delivery workers and snapshot
513531
builds. Fanout waits for worker capacity before admitting each recipient, so a
514532
fleet-wide change cannot create a fleet-sized task backlog or saturate the store
515533
and credential backends. Snapshot construction has a deadline that starts once
516534
a build holds a permit, and the gateway rejects encoded stream messages that
517-
approach the transport decoder limit. A later migration will apply these
518-
payloads directly and acknowledge their exact revisions before removing
519-
supervisor polling. At that point, the gateway will require a valid bootstrap
520-
before marking a session ready.
535+
approach the transport decoder limit. Durable apply operations and final polling
536+
removal remain separate follow-up work.
521537

522538
## Policy Revision Acknowledgement
523539

‎crates/openshell-core/src/grpc_client.rs‎

Lines changed: 116 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -960,6 +960,29 @@ fn settings_poll_result(inner: crate::proto::GetSandboxConfigResponse) -> Settin
960960
}
961961
}
962962

963+
impl From<crate::proto::SandboxConfigSnapshot> for SettingsPollResult {
964+
fn from(inner: crate::proto::SandboxConfigSnapshot) -> Self {
965+
Self {
966+
policy: inner.policy,
967+
version: inner.version,
968+
policy_hash: inner.policy_hash,
969+
config_revision: inner.config_revision,
970+
policy_source: PolicySource::try_from(inner.policy_source)
971+
.unwrap_or(PolicySource::Unspecified),
972+
settings: inner.settings,
973+
global_policy_version: inner.global_policy_version,
974+
provider_env_revision: inner.provider_env_revision,
975+
supervisor_middleware_services: inner.supervisor_middleware_services,
976+
workspace: inner.workspace,
977+
policy_validation_failure_mode: inner
978+
.policy_validation_failure_mode
979+
.parse()
980+
.unwrap_or_default(),
981+
extension_authentication_enabled: inner.extension_authentication_enabled,
982+
}
983+
}
984+
}
985+
963986
#[cfg(test)]
964987
mod settings_poll_tests {
965988
use super::settings_poll_result;
@@ -1001,6 +1024,25 @@ mod settings_poll_tests {
10011024
let legacy = settings_poll_result(GetSandboxConfigResponse::default());
10021025
assert!(!legacy.extension_authentication_enabled);
10031026
}
1027+
1028+
#[test]
1029+
fn delivered_sandbox_snapshot_uses_the_polling_projection() {
1030+
let result = super::SettingsPollResult::from(crate::proto::SandboxConfigSnapshot {
1031+
version: 7,
1032+
policy_hash: "hash-7".to_string(),
1033+
config_revision: 42,
1034+
provider_env_revision: 9,
1035+
workspace: "workspace-a".to_string(),
1036+
extension_authentication_enabled: true,
1037+
..Default::default()
1038+
});
1039+
assert_eq!(result.version, 7);
1040+
assert_eq!(result.policy_hash, "hash-7");
1041+
assert_eq!(result.config_revision, 42);
1042+
assert_eq!(result.provider_env_revision, 9);
1043+
assert_eq!(result.workspace, "workspace-a");
1044+
assert!(result.extension_authentication_enabled);
1045+
}
10041046
}
10051047

10061048
pub struct ProviderEnvironmentResult {
@@ -1012,6 +1054,80 @@ pub struct ProviderEnvironmentResult {
10121054
pub non_secret_environment_keys: Vec<String>,
10131055
}
10141056

1057+
impl From<crate::proto::ProviderEnvironmentSnapshot> for ProviderEnvironmentResult {
1058+
fn from(snapshot: crate::proto::ProviderEnvironmentSnapshot) -> Self {
1059+
use crate::proto::ProviderEnvironmentValueClassification;
1060+
1061+
let mut environment = HashMap::with_capacity(snapshot.values.len());
1062+
let mut credential_expires_at_ms = HashMap::new();
1063+
let mut static_credential_bindings = HashMap::new();
1064+
let mut non_secret_environment_keys = Vec::new();
1065+
for value in snapshot.values {
1066+
environment.insert(value.name.clone(), value.value);
1067+
if let Some(expires_at_ms) = value.expires_at_ms {
1068+
credential_expires_at_ms.insert(value.name.clone(), expires_at_ms);
1069+
}
1070+
match ProviderEnvironmentValueClassification::try_from(value.classification)
1071+
.unwrap_or_default()
1072+
{
1073+
ProviderEnvironmentValueClassification::NonSecret => {
1074+
non_secret_environment_keys.push(value.name);
1075+
}
1076+
ProviderEnvironmentValueClassification::StaticCredential => {
1077+
if let Some(binding) = value.static_credential_binding {
1078+
static_credential_bindings.insert(value.name, binding);
1079+
}
1080+
}
1081+
ProviderEnvironmentValueClassification::Unspecified => {}
1082+
}
1083+
}
1084+
Self {
1085+
environment,
1086+
provider_env_revision: snapshot.provider_env_revision,
1087+
credential_expires_at_ms,
1088+
dynamic_credentials: snapshot.dynamic_credentials,
1089+
static_credential_bindings,
1090+
non_secret_environment_keys,
1091+
}
1092+
}
1093+
}
1094+
1095+
#[cfg(test)]
1096+
mod provider_snapshot_tests {
1097+
use super::ProviderEnvironmentResult;
1098+
use crate::proto::{
1099+
ProviderEnvironmentSnapshot, ProviderEnvironmentValue,
1100+
ProviderEnvironmentValueClassification, StaticCredentialBinding,
1101+
};
1102+
1103+
#[test]
1104+
fn delivered_provider_snapshot_preserves_secret_classification_metadata() {
1105+
let result = ProviderEnvironmentResult::from(ProviderEnvironmentSnapshot {
1106+
provider_env_revision: 11,
1107+
values: vec![
1108+
ProviderEnvironmentValue {
1109+
name: "REGION".to_string(),
1110+
value: "west".to_string(),
1111+
classification: ProviderEnvironmentValueClassification::NonSecret.into(),
1112+
..Default::default()
1113+
},
1114+
ProviderEnvironmentValue {
1115+
name: "TOKEN".to_string(),
1116+
value: "redacted".to_string(),
1117+
classification: ProviderEnvironmentValueClassification::StaticCredential.into(),
1118+
static_credential_binding: Some(StaticCredentialBinding::default()),
1119+
..Default::default()
1120+
},
1121+
],
1122+
..Default::default()
1123+
});
1124+
assert_eq!(result.provider_env_revision, 11);
1125+
assert_eq!(result.environment.len(), 2);
1126+
assert_eq!(result.non_secret_environment_keys, ["REGION"]);
1127+
assert!(result.static_credential_bindings.contains_key("TOKEN"));
1128+
}
1129+
}
1130+
10151131
pub struct ProviderSubjectTokenExchangeResult {
10161132
pub access_token: String,
10171133
pub expires_in: i64,

‎crates/openshell-core/src/proto/mod.rs‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -79,7 +79,11 @@ pub use test::ObjectForTest;
7979
/// The supervisor stream is an internal, version-locked deployment contract.
8080
/// Bump this when either peer can no longer honor the previous stream
8181
/// semantics.
82-
pub const SUPERVISOR_PROTOCOL_REVISION: u32 = 1;
82+
pub const SUPERVISOR_PROTOCOL_REVISION: u32 = 2;
83+
84+
/// Stage 1 peers understand snapshot envelopes but do not apply them. They
85+
/// remain compatible while polling is retained for the rollout.
86+
pub const PREVIOUS_SUPERVISOR_PROTOCOL_REVISION: u32 = 1;
8387

8488
/// Revision implied by peers built before the handshake existed. Proto3 leaves
8589
/// the field unset, so such peers report zero.

0 commit comments

Comments
 (0)