Skip to content

cheduler: printedLog map is never pruned when nodes are deleted #2937

Description

@im-Toqeer-506

What happened

RegisterFromNodeAnnotations creates a printedLog map and passes it to every register() call:

printedLog := map[string]bool{}
for {
    ...
    s.register(labelSelector, printedLog)
}

register() adds a node to this map the first time it logs the node:

if printedLog[val.Name] {
    klog.V(5).InfoS("Node device updated", ...)
} else {
    klog.InfoS("Node device added", ...)
    printedLog[val.Name] = true
}

The problem is that entries are never removed.

onDelNode already cleans up other per-node state when a node is deleted, but it cannot clean printedLog because the map is local to RegisterFromNodeAnnotations.

Why this is a problem

  1. printedLog keeps growing for the lifetime of the scheduler. Each node name that is ever registered remains in the map.
  2. If a node is deleted and later recreated with the same name, the scheduler treats it as already seen and logs Node device updated at V(5) instead of the expected info-level Node device added.

This can make it harder for operators to see that a replacement node has registered.

Expected behavior

printedLog should be cleaned up when a node is deleted, just like the other per-node state.

How to reproduce

  1. Run hami-scheduler with the default log verbosity.
  2. Register a GPU node and confirm that Node device added is logged.
  3. Delete the node.
  4. Recreate a node with the same name.
  5. When it registers again, Node device added is not logged. Instead, it is logged as Node device updated at V(5).

Suggested fix

Move printedLog into the scheduler state and clean it up when the node is removed:

  • Add printedLog map[string]bool to the scheduler and initialize it in NewScheduler().
  • Remove the printedLog parameter from register().
  • Delete the node from s.printedLog in cleanupNodeUsage(), which is already called by onDelNode.

This keeps the change limited to pkg/scheduler/scheduler.go and reuses the existing per-node cleanup path.

Testing

Add a regression test covering:

node added → node deleted → same node name added again

The test should pass with -race.

Scope

Only pkg/scheduler/scheduler.go and pkg/scheduler/scheduler_test.go are affected. No API, annotation, or configuration changes are needed.

I'm happy to take this if the approach looks reasonable.

Metadata

Metadata

Assignees

Labels

kind/bugSomething isn't working

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions