Preserve pre-existing node cordons across driver upgrades

[ben-feldstein]

Fixes #212.

Problem

driver-manager uncordons the node unconditionally on its success path, in cleanupOnFailure(), and on the skip-eviction path that runs after a node reboot. It never checks whether the node was already cordoned before it acted. Any cordon placed by an administrator or another controller is therefore wiped whenever the driver pod is recreated, and on every node reboot. On the reboot path this is worst: driver-manager uncordons the node without ever having cordoned it.

The upgrade controller in k8s-operator-libs already solves this with a node-initial-state.unschedulable annotation and skips the uncordon for nodes that were cordoned before the upgrade started. driver-manager has no equivalent.

Fix

I applied the same pattern to driver-manager.

• CordonNode first records the node's current schedulable state in a nvidia.com/driver-manager.node-initial-state.unschedulable annotation, then cordons. The annotation is written before the cordon and only when absent, so a restart of the init container after the cordon cannot overwrite the state recorded by the first run.
• UncordonNode restores the recorded state instead of blindly uncordoning: it uncordons only when the annotation records false, skips the uncordon when it records true, and removes the annotation in both cases. When the annotation is absent, driver-manager never cordoned the node in any run, so the node is left untouched.
• On an uncordon failure the annotation is kept, so a retry after the init container restarts can still uncordon.
• A recording found on a schedulable node is treated as stale and overwritten: a cycle interrupted after our cordon would have left the node unschedulable, so the annotation can only have survived a failed removal.
• The shouldSkipUninstall early return now also runs the state-aware uncordon, so a cordon left by a crashed prior cycle is released and the annotation cleaned up. With the annotation absent this is a no-op.

Scenario	Before	After
Node not cordoned, driver upgrade succeeds	uncordoned	uncordoned (unchanged)
Node not cordoned, drain fails	uncordoned, retry loop flaps the cordon	uncordoned, retry loop flaps the cordon (unchanged)
Node manually cordoned before upgrade	uncordoned	stays cordoned
Node manually cordoned, drain fails	uncordoned	stays cordoned
Node reboot with a manual cordon in place	uncordoned	stays cordoned
Init container restarts between cordon and uncordon	uncordoned	uncordoned, recorded state survives the restart

Notes

• One-time rollout note: a node that a previous driver-manager release cordoned mid-cycle carries no annotation, so this version will not auto-uncordon it. Operators should uncordon such nodes manually once after upgrading.
• A cordon placed by an external actor during the upgrade window is still restored to the recorded initial state at the end of the cycle. The same race exists in the upgrade controller.

Testing

• Added unit tests for CordonNode/UncordonNode against a fake clientset covering: cordon/uncordon round trip on a schedulable node, a pre-cordoned node staying cordoned through the uncordon, the recorded state surviving an init container restart, a stale recording on a schedulable node being reconciled, an uncordon with no prior cordon leaving an external cordon in place, and the recording never being overwritten while the node is unschedulable.
• go build ./..., go vet ./... (linux), and go test ./... all pass.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[kvalliyurnatt]

Thanks for your contribution. Changes look mostly good, I think there is one thing that needs to be addressed

[kvalliyurnatt]

If uncordoning succeeds but annotation removal fails, a stale "false" annotation remains. If an administrator later cordons the node, the next UncordonNode may interpret that stale annotation as ownership and remove the administrator’s cordon. Could the uncordon and annotation removal be performed atomically in one Kubernetes patch?