Skip to content

README.md

Latest commit

 

History

History
79 lines (57 loc) · 2.52 KB

README.md

File metadata and controls

79 lines (57 loc) · 2.52 KB
Raw

k8s-redis-workload-scaler

Kubernetes operator that scales a Deployment from Redis Stream consumer-group backlog (pending + lag), using scaling logic in autoscaler-engine.

What it does

  • Watches RedisStreamScaler CRs and the target Deployment
  • Reads backlog via Redis XINFO GROUPS
  • Sets Deployment.spec.replicas with min/max, tasksPerPod, and scale-down cooldown after scale-up
  • Leader election enabled by default (single active reconciler)

How scaling works

Each reconcile (on a timer and when the CR or target Deployment changes):

  1. Backlog — XINFO GROUPS on streamName; for groupName, backlog = pending + lag (unclaimed messages plus messages not yet delivered to consumers).
  2. Desired replicas — ceil(backlog / tasksPerPod), then clamped to minReplicas … maxReplicas.
  3. Apply — patch Deployment.spec.replicas when desired differs from current.

Scale-up happens as soon as backlog needs more pods. Scale-down is delayed: for cooldownPeriod after the last scale-up, the operator will not reduce replicas even if backlog drops (scale-down is allowed once that window passes). That asymmetry gives workers time to finish in-flight work; pair it with a sensible terminationGracePeriodSeconds on the worker Deployment.

Scaling logic lives in autoscaler-engine; this operator wires it to Kubernetes and Redis. Do not run HPA on the same Deployment.

Project layout

api/v1alpha1/          CRD types (+ kubebuilder markers)
internal/controller/   Reconciler and Redis/Deployment adapters
cmd/main.go            Manager entrypoint
config/                Kustomize (CRD, RBAC, manager, default bundle)
docs/deploy.md         Deploy to rbpi
hack/boilerplate.go.txt  License header for make generate

Development

make manifests generate
make test
make lint

Deploy

export KUBECONFIG=/path/to/alesrpi-lab-iac/kubeconfig.yaml
make deploy-rbpi

Details: docs/deploy.md.

Scale test

With operator, Redis, worker Deployment, and the sample CR applied:

make scale-load
make scale-observe
make scale-load-clear

RedisStreamScaler example

spec:
  scaleTargetRef: worker-deployment
  redisAddr: redis.default.svc.cluster.local:6379
  streamName: events
  groupName: workers
  minReplicas: 1
  maxReplicas: 5
  tasksPerPod: 10
  cooldownPeriod: 30s
  pollInterval: 5s

License

Apache License 2.0. See LICENSE.