fix: skip unreachable RedisReplication pods during role discovery

[leeworker]

Fixes #1842

Summary

When a worker node crashes, Redis pods can remain Running/Ready with stale Kubernetes status while Redis is unreachable. GetRedisNodesByRole previously fail-fast on the first probe error, which blocked the entire RedisReplication reconcile loop and prevented healthy pods from updating status.masterNode and redis-role labels.

This change:

• Skips per-pod Redis probe failures in GetRedisNodesByRole (log + continue) instead of aborting reconcile
• Removes stale redis-role labels from pods that are probeable in Kubernetes but unreachable over Redis, so master Services do not keep routing to dead pods during the stale-Ready window

Test plan

• go test ./internal/k8sutils/... -run 'GetRedisNodesByRole|IsRedisPodProbeable'
• go test ./internal/controller/common/redis/... -run UpdateRedisRoleLabel
• go test ./internal/controller/redisreplication/... -run 'TestReconcile'

Made with Cursor

[leeworker]

Hi maintainers,

Could you please approve the workflows when you have a chance?

This PR fixes #1842: when a worker node crashes, Redis pods can remain Running/Ready with stale status while Redis is unreachable. GetRedisNodesByRole previously fail-fast on the first probe error and blocked the whole RedisReplication reconcile loop, so healthy pods could not update status.masterNode or redis-role labels.

Thanks!