[Feature]: Add a label-based remediation provider to Janitor

[suket22]

Prerequisites

• I searched existing issues

Code of Conduct

• I agree to follow NVSentinel's Code of Conduct

Feature Summary

The existing set of janitor-provider(s) currently supports either the cloud
provider APIs (aws, gcp, azure, oci, nebius), or performs the reboots on the host directly via a privileged pod. There's no way for NVSentinel to delegate the remediation action to the platform it's running on.

Concretely, if you were running NVSentinel in an on-prem bare metal environment where the actions aren't directly exposed via API, we've got limited options for remediation. If we add a label provider that implements model.CSPClient by writing node labels and observing completion, then we could have any infrastructure provider take over the remediation actions natively.

Problem/Use Case

The problem I'm solving is to decouple the infrastructure provider from Janitor's remediation action. So if you wanted Janitor to function against OpenStack provisioned nodes, NICo, EC2 etc, you could do so by applying a label and letting an external system watch the K8s API (the one true common data surface) and act on it. This way Janitor never needs to be fed platform credentials either since K8s creds are sufficient.

Proposed Solution

Proposed behaviour

SendRebootSignal — patch the configured reboot label onto the Node with the
configured reason value.

IsNodeReady — report completion when the node has actually rebooted.
We could compare node.Status.NodeInfo.BootID against the pre-reboot value as that's what generic does I think too.

SendTerminateSignal — patch the configured terminate label. Completion is the node object being removed.

Other things -

• We should probably handle cases where we time out if the external system doesn't act in a configured amount of time - maybe emitting a K8s event or so in this case?
• The actual labels and annotation keys being put on the nodes should be configurable as every external system may want to look at something different.

Component

Plugins