Skip to content

Add node management APIs: drain/undrain, taint/untaint, labels and automatic workload relocation - #25

Merged
miladhzzzz merged 2 commits into
mainfrom
codex/add-node-drain,-taint,-and-management-commands
Jul 1, 2026
Merged

miladhzzzz merged 2 commits into
mainfrom
codex/add-node-drain,-taint,-and-management-commands

Conversation

@miladhzzzz

Copy link
Copy Markdown
Member

Motivation

  • Provide operator-facing control-plane APIs to manage node lifecycle (drain/undrain), taints, and labels so nodes can be taken out of scheduling without manual etcd edits or relying on heartbeat-only flags.
  • Ensure scheduler enforces taints and drains by excluding tainted/draining nodes from placement and relocating workloads automatically when an operator drains a node.
  • Surface node taints and label changes through the control API so the gateway and clients can observe node metadata consistently.

Description

  • Added new control RPCs to the scheduler AgentControl API and proto: DrainNode, UndrainNode, TaintNode, UntaintNode, SetNodeLabel, and DeleteNodeLabel, plus message types for taints, drain responses, and label operations in api/proto/control.proto.
  • Persisted taints on the scheduler Node model and exposed them in NodeView, plus regenerated protobuf/gRPC stubs used by scheduler, gateway, and shared pkg clients.
  • Implemented scheduler-side node management methods (MarkNodeDraining, MarkNodeReady, TaintNode, UntaintNode, SetNodeLabel, DeleteNodeLabel) and an updateNode helper that persists changes to etcd and updates the in-memory cache.
  • Preserved Draining status during heartbeats and made heartbeat responses set drain_node=true when the node is in Draining state.
  • Excluded nodes with NoSchedule-style taints from selectNodeForWorkload() and added RelocateWorkloadsFromNode() which automatically reassigns workloads when a node is marked Draining.
  • Added gRPC handlers in the scheduler (internal/grpcapi/service.go) and gateway proxy methods/controllers (persys-gateway/services/*, controllers/*, routes/*) plus HTTP routes and JSON handlers for the new operations.

Testing

  • Generated updated protobuf stubs with make proto in persys-scheduler and persys-gateway and regenerated dependent internal/controlv1 and pkg/scheduler/controlv1 files successfully.
  • Ran unit/test targets: cd persys-scheduler && go test ./internal/controlv1 and cd persys-gateway && go test ./internal/controlv1, both completed without errors.
  • Exercised additional package test runs (no-test-files suites) via go test across internal/grpcapi, internal/scheduler, and gateway controller/service/routes packages which returned successfully (no failing tests).
  • Verified gofmt and git diff --check validations as part of the change workflow with no formatting or lint errors reported.

Codex Task

@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

@miladhzzzz miladhzzzz added the enhancement New feature or request label Jun 25, 2026
@miladhzzzz miladhzzzz self-assigned this Jun 25, 2026
@miladhzzzz
miladhzzzz requested a review from parmisk80 June 30, 2026 11:06

@miladhzzzz miladhzzzz left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/LGTM

@miladhzzzz
miladhzzzz merged commit 5403281 into main Jul 1, 2026
3 of 4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

codex enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant