persys-dev · miladhzzzz · Sep 7, 2026 · Sep 7, 2026 · Sep 7, 2026 · Sep 7, 2026
diff --git a/persys-scheduler/CHANGELOG.md b/persys-scheduler/CHANGELOG.md
@@ -1,5 +1,67 @@
 # Changelog
 
+## 2026-07-28 (Unreleased)
+
+Source: 1000+ node scaling initiative (`persys-scheduler-1000-node-scaling-plan.md`)
+
+### Summary
+
+This release targets running persys-scheduler at 1,000-5,000+ node fleets and behind multiple replicas. It cuts agent-facing fan-out from O(workloads) to O(nodes), adds etcd compare-and-swap to every concurrent write path, adds a watch-backed node cache for placement, adds leader election with an optional active-active sharding mode, replaces single-factor node scoring with a weighted placement algorithm that accounts for in-flight (not-yet-heartbeated) resource commitments, and fixes CoreDNS self-registration so persys-gateway can discover more than one running replica at a time.
+
+### Major Features
+
+1. **Agent connection pooling**
+   - Replaced per-RPC dial-then-close with a pooled `map[nodeID]*grpc.ClientConn`, keepalive-checked and reused across calls
+   - Removes a full TLS handshake from every single agent RPC — previously paid on every apply/delete/status/list call, to every node, every cycle
+
+2. **Reconciliation fan-out: O(workloads) → O(nodes)**
+   - The reconciler now fetches each node's full workload list once per cycle (via the batch RPC drift-detection already used) instead of one status RPC per workload
+   - Bounded-concurrency workload processing (`SCHEDULER_RECONCILE_CONCURRENCY`, default 64), plus a cycle-overlap guard so a slow cycle can't stack with the next tick
+   - `MonitorNodes`, `MonitorWorkloads`, and `detectDriftOnce` got the same bounded-concurrency treatment
+
+3. **etcd compare-and-swap on all concurrent write paths**
+   - New `RetryableEtcdCASPut`; every node-mutating function (heartbeat, drain/ready/taint/label/capability updates, NotReady transitions) and every workload-status-mutating function (`UpdateWorkloadStatus`, `UpdateWorkloadLogs`, `UpdateWorkloadMetadata`, `UpdateWorkloadRuntimeDetails`) now retries against a fresh read on conflict instead of silently overwriting a concurrent writer
+   - Removed a dead `/retries/{id}` write nothing ever read; fixed a bug where usage telemetry (`Usage`) was silently dropped from the etcd status projection on every read
+
+4. **Watch-backed live node cache for placement**
+   - New `node_watch.go`: full resync then a live etcd `Watch` keep the in-memory node cache current
+   - `selectNodeForWorkload` reads from it, falling back to a live scan only if the cache isn't populated yet (startup, or mid-resync)
+
+5. **Leader election with failover / active-active HA modes** (new env: `SCHEDULER_HA_MODE`, `SCHEDULER_SHARD_COUNT`, `SCHEDULER_SHARD_INDEX`)
+   - New `leader.go`: etcd-lease-based election (`go.etcd.io/etcd/client/v3/concurrency`) so multiple scheduler replicas can run against the same etcd cluster with automatic failover
+   - `SCHEDULER_HA_MODE=failover` (default): exactly one replica active cluster-wide; the others are hot standbys that take over automatically if it dies
+   - `SCHEDULER_HA_MODE=active-active`: nodes are partitioned across `SCHEDULER_SHARD_COUNT` shards by a stable hash of node ID; each replica only drives reconciliation/monitoring/drift-detection for the nodes in its `SCHEDULER_SHARD_INDEX`, so multiple shards run concurrently — run more than one replica per shard index for HA within a shard
+   - The gRPC API (`RegisterNode`, `Heartbeat`, `ApplyWorkload`, ...) runs unconditionally on every replica in both modes, since those paths are CAS-protected; only the singleton convergence loops are gated
+
+6. **Weighted placement algorithm with in-flight resource reservation**
+   - Replaced single-factor "lowest CPU+memory average" sorting with a weighted score: CPU headroom, memory headroom, and a spread term (workload count relative to the busiest candidate), plus a small deterministic tie-breaker so exact ties don't always resolve to the same node
+   - New in-flight reservation tracking (`placement.go`): resources committed to a just-assigned workload are counted against its node immediately, before that node's next heartbeat reflects the change — closes a real oversubscription window that gets materially more likely now that reconciliation, monitoring, and (in active-active mode) multiple scheduler replicas can all be placing/converging workloads concurrently
+
+7. **CoreDNS multi-replica fix**
+   - Scheduler self-registration (used by persys-gateway to discover the scheduler) previously wrote to one fixed etcd key; every replica overwrote the others on startup, so the gateway could only ever resolve one replica regardless of how many were running
+   - Now keyed per-instance (mirroring the pattern already used for agent node registration), so CoreDNS returns one record per running replica; added deregistration on clean shutdown so a stopped replica doesn't linger as a dead record until its TTL expires
+   - Note: this DNS mechanism is for **gateway → scheduler** discovery only; agents do not use CoreDNS to reach the scheduler
+
+### Deployment note
+
+Running multiple scheduler replicas under plain `docker compose up` (not Swarm) requires removing the scheduler's own host port publishing and putting a TCP-passthrough load balancer (e.g. HAProxy) in front instead — otherwise replicas past the first fail to bind the same host port. See `docker-compose.scheduler-ha.snippet.yml` and `haproxy.cfg`. Not needed under `docker stack deploy` (Swarm), where the ingress routing mesh already handles this.
+
+### Breaking Changes
+
+None. `SCHEDULER_HA_MODE` defaults to `failover`, which preserves prior single-active-instance behavior exactly. All other new env vars have defaults matching prior behavior (`SCHEDULER_RECONCILE_CONCURRENCY=64`, `SCHEDULER_SHARD_COUNT=1`, `SCHEDULER_SHARD_INDEX=0`).
+
+### Known Limitations
+
+- Active-active mode has no explicit shard hand-off protocol: if a node fails and its workloads are reassigned to a node owned by a different shard, there's a brief window (expected to self-heal within one reconcile interval) where neither shard is actively driving that workload. See `sharding.go` for details.
+- Redis remains a single instance with no Sentinel/cluster failover.
+- Not load-tested at target scale in this round; reasoning is from code inspection, not measurement.
+
+### Changed Files
+
+See `persys-scheduler-scaling-fixes.patch` for the full diff. New files: `internal/scheduler/leader.go`, `internal/scheduler/sharding.go`, `internal/scheduler/node_watch.go`, `internal/scheduler/placement.go`.
+
+---
+
 ## 2026-05-29 (Unreleased)
 
 Source: `git diff -- persys-scheduler`

diff --git a/persys-scheduler/README.md b/persys-scheduler/README.md
@@ -6,8 +6,9 @@ It accepts node registrations, stores cluster state in etcd, places workloads on
 ## What This Service Does
 
 - Exposes a gRPC control API for nodes and workload lifecycle.
-- Persists scheduler state in etcd (`/nodes`, `/workloads-spec`, `/workloads-status`, `/volumes`, `/attachments`, assignments, retries, reconciliation records, events).
-- Offloads high-churn telemetry data to Redis for automatic cleanup (reconciliation metadata, event logs).
+- Persists scheduler state in etcd (`/nodes`, `/workloads-spec`, `/workloads-status`, `/volumes`, `/attachments`, assignments, retries, reconciliation records).
+- Emits cluster-wide events (node lost, workload scheduled/failed, drift detected, retries, ...) to a Redis Stream — not etcd — for exactly the reasons under "Cluster Events" below.
+- Offloads high-churn telemetry data to Redis for automatic cleanup (reconciliation metadata, event history).
 - Schedules workloads based on node readiness, resources, labels, supported workload types, and storage driver capabilities.
 - Reconciles workloads (`Running` / `Stopped` / `Deleted`) against agent-reported state with exponential backoff protection.
 - Manages workload retry state with failure grace periods to allow transient failures to self-heal.
@@ -24,13 +25,56 @@ flowchart LR
     AG[Compute Agents] -->|RegisterNode + Heartbeat| SCH
     SCH -->|Apply/Delete/Get/ListWorkloads| AG
 
-    SCH -->|State + Assignments + Events + Drift Marks| ETCD[(etcd)]
+    SCH -->|State + Assignments + Drift Marks| ETCD[(etcd)]
+    SCH -->|Events + Reconciliation Telemetry| REDIS[(Redis)]
     SCH -->|A/SRV records| DNS[(CoreDNS)]
 
     SCH -->|/metrics| PROM[(Prometheus)]
     SCH -->|OTLP traces| OTLP[(Jaeger/OTel Collector)]
 ```
 
+## High Availability and Sharding
+
+Multiple scheduler replicas can run against the same etcd cluster. `SCHEDULER_HA_MODE` controls how they coordinate:
+
+- **`failover`** (default): all replicas contend for a single etcd-lease-based election. Exactly one is ever active — driving reconciliation, node/workload monitoring, drift detection, and the placement node-cache watch. The others are hot standbys; if the active replica dies or its lease expires (crash, GC pause past the lease TTL, network partition), another takes over automatically, typically within one lease TTL (15s by default).
+- **`active-active`**: nodes are partitioned across `SCHEDULER_SHARD_COUNT` shards by a stable hash of node ID. Each replica is assigned a `SCHEDULER_SHARD_INDEX` and only drives convergence for the nodes that hash into it — so multiple shards make progress concurrently instead of one replica doing all the work. Run more than one replica per shard index to get failover *within* a shard.
+
+In both modes, the gRPC API (`RegisterNode`, `Heartbeat`, `ApplyWorkload`, `DeleteWorkload`, ...) runs unconditionally on **every** replica — those write paths are protected by etcd compare-and-swap, so it's safe for an external load balancer or Swarm's ingress mesh to route agent traffic to any replica regardless of which one currently holds an election. Only the background convergence loops are gated.
+
+**Known limitation of `active-active` mode**: there's no explicit hand-off protocol between shards. If a node fails and its workloads are reassigned (`RelocateWorkloadsFromNode`) to a node owned by a different shard, that shard picks up ownership on its next cycle automatically — but there's a brief window (expected to self-heal within one reconcile interval) where neither shard is actively driving that specific workload. This is a scheduling-latency gap, not a correctness bug, but worth knowing before relying on `active-active` mode for latency-sensitive failover.
+
+Deploying more than one replica without Docker Swarm requires a TCP-passthrough load balancer in front (see `docker-compose.scheduler-ha.snippet.yml` / `haproxy.cfg` in the repo) — plain `docker compose up` can't have multiple containers of the same service all bind the same host port. This isn't needed under `docker stack deploy`, where Swarm's own ingress routing mesh already handles it.
+
+## Object storage (Ceph RGW)
+
+Object buckets are **not** stored in etcd. The scheduler proxies **Ceph RGW** (S3 API):
+
+| Operation | Backend |
+|-----------|---------|
+| List / create / delete bucket | RGW |
+| List objects | RGW |
+| User access key / secret | **Vault KV** (path `secret/data/persys/rgw/buckets/{name}` by default) |
+
+- Control-plane RGW credentials: `PERSYS_RGW_ENDPOINT`, `PERSYS_RGW_ACCESS_KEY`, `PERSYS_RGW_SECRET_KEY`
+- Vault auth: **vault-manager** `GetServiceCredentials(PERSYS_VAULT_SERVICE_NAME)` → AppRole login (same pattern as certmanager). Optional `PERSYS_VAULT_TOKEN` for break-glass only.
+- Optional `PERSYS_RGW_ADMIN_PATH` (default `admin`) registers generated keys via RGW Admin Ops so S3 clients can authenticate.
+- gRPC: `CreateBucket`, `ListBuckets`, `GetBucket`, `DeleteBucket`, `GetBucketAccess`, `ListBucketObjects`
+- REST (via gateway): `/buckets`, `/buckets/:id`, `/buckets/:id/access`, `/buckets/:id/objects`
+- **Agents are not involved** in object storage.
+
+Standalone **block disks** remain a separate control-plane surface (`/disks`, hard pinning for local attach).
+
+## Placement
+
+`selectNodeForWorkload` filters candidate nodes for feasibility (status, taints, workload-type capability, storage driver, CPU/memory availability), then scores the survivors and picks the highest score. The score blends:
+
+- CPU and memory headroom, adjusted for resources already committed to workloads assigned earlier in the same scheduling window but not yet reflected in that node's own heartbeat-reported availability (see "in-flight reservations" below).
+- A spread term based on how many workloads are already on the node relative to the busiest candidate, so nodes with similar CPU/memory ratios aren't treated as identical if one is already hosting far more workloads.
+- A small deterministic tie-breaker (hashed from workload + node ID) so exact ties — common in a homogeneous fleet — don't always resolve to the same node.
+
+**In-flight reservations**: a node's `AvailableCPU`/`AvailableMemory` only update via that node's own heartbeat, which lags behind a placement decision. The scheduler tracks recently-assigned-but-not-yet-heartbeat-confirmed commitments in memory and subtracts them from a node's effective headroom during scoring (self-expiring after 90s), so concurrent placement decisions — routine now that reconciliation runs with bounded concurrency and `active-active` mode can have multiple replicas placing workloads simultaneously — don't all pick the same "least loaded" node and oversubscribe it before any of their heartbeats catch up.
+
 ## Operating Modes
 
 The scheduler now runs with explicit operating modes:
@@ -92,19 +136,26 @@ The scheduler uses Redis to store high-churn telemetry data, significantly reduc
 ### What Gets Stored in Redis
 
 - Reconciliation metadata (per-workload retry attempt tracking, backoff timers)
-- Event history (bounded list with TTL and max entries)
+- Cluster-wide event history (Redis Stream — see "Cluster Events" below)
 - Optionally, high-frequency reconciliation status updates
 
 ### Data Retention
 
 - Reconciliation data: TTL 24 hours (configurable via `REDIS_RECONCILE_TTL`)
-- Event history: TTL 24 hours (configurable via `REDIS_EVENT_TTL`)
-- Maximum event entries: 1000 (configurable via `REDIS_EVENT_MAX_ENTRIES`)
+- Event history: TTL 24 hours, refreshed on every write (configurable via `REDIS_EVENT_TTL`); capped at 1000 entries via approximate stream trimming (configurable via `REDIS_EVENT_MAX_ENTRIES`)
 
 ### Graceful Fallback
 
-- If Redis is unavailable, scheduler automatically falls back to etcd for all storage
-- Scheduler continues operating normally with etcd-only mode
+- Reconciliation metadata falls back to etcd if Redis is unavailable.
+- Cluster events do **not** fall back to etcd — see "Cluster Events" below for why. If Redis is unavailable, events are dropped (logged) rather than persisted elsewhere, and `ListEvents`/`WatchEvents` callers see an empty result rather than an error.
+
+## Cluster Events
+
+`emitEvent` records cluster-wide, human-readable events — `NodeJoined`, `NodeLost`, `NodeLeft`, `WorkloadScheduled`, `WorkloadFailed`, `DriftDetected`, `RetryTriggered`, `Rescheduled`, `Relocated` — for consumption by `persysctl` and dashboard UIs via the `ListEvents` (recent history) and `WatchEvents` (live tail, replays recent history first) gRPC RPCs, both supporting optional filtering by `type`/`workload_id`/`node_id`.
+
+Events live **only in Redis** (a single shared Stream, `scheduler:events`), not etcd. All scheduler replicas share the same Redis instance, so this is cluster-wide in exactly the same sense etcd-backed state is — an event emitted by whichever replica handled the triggering request is visible to every replica's `WatchEvents` callers.
+
+This is a deliberate choice, not an oversight: events are high-churn and purely observability-oriented, so they shouldn't compete for etcd's write throughput with heartbeats and CAS-retried reconciliation — especially since event volume tends to spike at exactly the moments (node flapping, mass retries during an incident) when etcd is already under the most load from everything else. Redis Streams also give bounded retention for free (`MAXLEN ~` trimming applied at write time) instead of needing a separate scan-and-delete sweep, which the previous etcd-backed version required since event IDs are random UUIDs with no cheap time-range key trick available.
 - No data loss or service interruption
 
 ### Storage Benefits
@@ -164,10 +215,11 @@ When reconciliation/apply fails, scheduler updates workload retry state:
 
 ## DNS and Service Discovery
 
-- Scheduler self-registers in CoreDNS on startup.
+- Scheduler self-registers in CoreDNS on startup, for discovery **by persys-gateway** — agents do not use CoreDNS to reach the scheduler (they connect via the address/LB they were configured with).
 - SRV record: `_persys-scheduler.<DOMAIN>`.
 - A record fallback: `persys-scheduler.<DOMAIN>`.
-- Agents register under shard-aware records: `<nodeID>.<SCHEDULER_SHARD_KEY>.agents.persys.cloud`.
+- Each replica registers under its own instance-keyed child rather than one shared key, so running multiple replicas produces one record per running replica instead of the last one to start silently overwriting the others. Deregisters its own record on clean shutdown.
+- Agents register under shard-aware records: `<nodeID>.<SCHEDULER_SHARD_KEY>.agents.persys.cloud`. (Note: `SCHEDULER_SHARD_KEY` here is a DNS namespace prefix for multi-environment segregation — unrelated to the reconciliation sharding described under High Availability below, despite the similar name.)
 - If CoreDNS is unavailable, scheduler logs a warning and continues running.
 
 ## API
@@ -272,13 +324,15 @@ mTLS:
 - `PERSYS_VAULT_ADDR` (default `http://127.0.0.1:8200`)
 - `PERSYS_VAULT_AUTH_METHOD` (`token` or `approle`)
 - `PERSYS_VAULT_TOKEN` (token auth)
-- `PERSYS_VAULT_APPROLE_ROLE_ID` / `PERSYS_VAULT_APPROLE_SECRET_ID` (AppRole auth)
+- `PERSYS_VAULT_MANAGER_ADDR` (default `vault-manager:50069`) — AppRole via vault-manager
+- `PERSYS_VAULT_APPROLE_ROLE_ID` / `PERSYS_VAULT_APPROLE_SECRET_ID` (optional legacy; prefer vault-manager)
 - `PERSYS_VAULT_PKI_MOUNT` (default `pki`)
 - `PERSYS_VAULT_PKI_ROLE` (default `persys-scheduler`)
 - `PERSYS_VAULT_CERT_TTL` (default `24h`)
 - `PERSYS_VAULT_RETRY_INTERVAL` (default `1m`)
 - `PERSYS_VAULT_SERVICE_NAME` (default `persys-scheduler`)
 - `PERSYS_VAULT_SERVICE_DOMAIN` (optional)
+- Object storage: `PERSYS_RGW_*` and Vault KV paths — see **Object storage (Ceph RGW)** above
 
 ## Workload Utilization Telemetry