Skip to content

Commit 4c555af

Browse files
authored
feat: add weightedRoundRobin load balancer algorithm (#4124)
1 parent d0679eb commit 4c555af

18 files changed

Lines changed: 599 additions & 4 deletions

File tree

.gitignore

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,3 +25,5 @@ skptesting/lorem.html
2525
.vscode/*
2626
*.test
2727

28+
skptesting/okserver
29+
skptesting/lb-algorithms.eskip

config/config.go

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -584,7 +584,7 @@ func NewConfig() *Config {
584584
flag.StringVar(&cfg.KubernetesValkeyServiceName, "kubernetes-valkey-service-name", "", "Sets name for valkey to be used to lookup endpoints")
585585
flag.IntVar(&cfg.KubernetesValkeyServicePort, "kubernetes-valkey-service-port", 6379, "Sets the port for valkey to be used to lookup endpoints")
586586
flag.StringVar(&cfg.KubernetesBackendTrafficAlgorithmString, "kubernetes-backend-traffic-algorithm", kubernetes.TrafficPredicateAlgorithm.String(), "sets the algorithm to be used for traffic splitting between backends: traffic-predicate or traffic-segment-predicate")
587-
flag.StringVar(&cfg.KubernetesDefaultLoadBalancerAlgorithm, "kubernetes-default-lb-algorithm", kubernetes.DefaultLoadBalancerAlgorithm, "sets the default algorithm to be used for load balancing between backend endpoints, available options: roundRobin, consistentHash, random, powerOfRandomNChoices")
587+
flag.StringVar(&cfg.KubernetesDefaultLoadBalancerAlgorithm, "kubernetes-default-lb-algorithm", kubernetes.DefaultLoadBalancerAlgorithm, "sets the default algorithm to be used for load balancing between backend endpoints, available options: roundRobin, consistentHash, random, powerOfRandomNChoices, weightedRoundRobin")
588588
flag.BoolVar(&cfg.KubernetesForceService, "kubernetes-force-service", false, "overrides default Skipper functionality and routes traffic using Kubernetes Services instead of Endpoints")
589589
flag.StringVar(&cfg.KubernetesStatusFromService, "kubernetes-status-from-service", "", "when set to <namespace>/<name>, updates Ingress status.loadBalancer.ingress from the referenced service")
590590

dataclients/kubernetes/deploy/apply/routegroups_crd.yaml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -67,11 +67,13 @@ spec:
6767
`random` - backend is chosen at random.
6868
`consistentHash` - backend is chosen by [consistent hashing](https://en.wikipedia.org/wiki/Consistent_hashing) algorithm based on the request key. The request key is derived from `X-Forwarded-For` header or request remote IP address as the fallback. Use [`consistentHashKey`](filters.md#consistenthashkey) filter to set the request key. Use [`consistentHashBalanceFactor`](filters.md#consistenthashbalancefactor) to prevent popular keys from overloading a single backend endpoint.
6969
`powerOfRandomNChoices` - backend is chosen by selecting N random endpoints and picking the one with least outstanding requests from them (see http://www.eecs.harvard.edu/~michaelm/postscripts/handbook2001.pdf).
70+
`weightedRoundRobin` - backend is chosen by smooth weighted round robin with dynamic weights based on the ratio of successful round trips per endpoint, weights are updated by the passive health check when enabled.
7071
enum:
7172
- roundRobin
7273
- random
7374
- consistentHash
7475
- powerOfRandomNChoices
76+
- weightedRoundRobin
7577
type: string
7678
endpoints:
7779
description: Endpoints is required for type `lb`

dataclients/kubernetes/kube.go

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -257,7 +257,7 @@ type Options struct {
257257
BackendTrafficAlgorithm BackendTrafficAlgorithm
258258

259259
// DefaultLoadBalancerAlgorithm sets the default algorithm to be used for load balancing between backend endpoints,
260-
// available options: roundRobin, consistentHash, random, powerOfRandomNChoices
260+
// available options: roundRobin, consistentHash, random, powerOfRandomNChoices, weightedRoundRobin
261261
DefaultLoadBalancerAlgorithm string
262262

263263
// ForwardBackendURL allows to use <forward> backend via kubernetes, for example routegroup backend `type: forward`.

docs/kubernetes/migrate.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -629,6 +629,7 @@ Available algorithms:
629629
- `random`
630630
- `consistentHash`
631631
- `powerOfRandomNChoices`
632+
- `weightedRoundRobin`
632633
633634
Your JIT based runtime applications have to ramp up slowly to traffic.
634635
You can use the [fadeIn](../reference/filters.md#fadein) filter to

docs/kubernetes/routegroup-crd.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -72,7 +72,7 @@ fields are the name and the type, while the rest of the fields may be required b
7272
name: <string>
7373
type: <string> one of "service|shunt|loopback|dynamic|lb|network|forward"
7474
address: <string> optional, required for type=network
75-
algorithm: <string> optional, valid for type=lb|service, values=roundRobin|random|consistentHash|powerOfRandomNChoices
75+
algorithm: <string> optional, valid for type=lb|service, values=roundRobin|random|consistentHash|powerOfRandomNChoices|weightedRoundRobin
7676
endpoints: <stringarray> optional, required for type=lb
7777
serviceName: <string> optional, required for type=service
7878
servicePort: <number> optional, required for type=service

docs/reference/backends.md

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -297,6 +297,7 @@ Current implemented algorithms:
297297
- `random`: backend is chosen at random
298298
- `consistentHash`: backend is chosen by [consistent hashing](https://en.wikipedia.org/wiki/Consistent_hashing) algorithm based on the request key. The request key is derived from `X-Forwarded-For` header or request remote IP address as the fallback. Use [`consistentHashKey`](filters.md#consistenthashkey) filter to set the request key. Use [`consistentHashBalanceFactor`](filters.md#consistenthashbalancefactor) to prevent popular keys from overloading a single backend endpoint.
299299
- `powerOfRandomNChoices`: backend is chosen by powerOfRandomNChoices algorithm with selecting N random endpoints and picking the one with least outstanding requests from them. (http://www.eecs.harvard.edu/~michaelm/postscripts/handbook2001.pdf)
300+
- `weightedRoundRobin`: backend is chosen by [smooth weighted round robin](https://github.com/nginx/nginx/commit/52327e0627f49dbda1e8db695e63a4b0af4448b1) with dynamic weights based on the ratio of successful round trips per endpoint. Weights are updated by the [passive health check](../operation/operation.md) stats loop, so they only change when the passive health check is enabled; with equal weights it behaves like `roundRobin`.
300301
- __TODO__: https://github.com/zalando/skipper/issues/557
301302
302303
Route example with 2 backends and the `roundRobin` algorithm:
@@ -319,6 +320,11 @@ Route example with 2 backends and the `powerOfRandomNChoices` algorithm:
319320
r0: * -> <powerOfRandomNChoices, "http://127.0.0.1:9998", "http://127.0.0.1:9997">;
320321
```
321322
323+
Route example with 2 backends and the `weightedRoundRobin` algorithm:
324+
```
325+
r0: * -> <weightedRoundRobin, "http://127.0.0.1:9998", "http://127.0.0.1:9997">;
326+
```
327+
322328
Proxy with `roundRobin` loadbalancer and two backends:
323329
```sh
324330
$ ./bin/skipper -inline-routes 'r0: * -> <roundRobin, "http://127.0.0.1:9998", "http://127.0.0.1:9997">;'

loadbalancer/algorithm.go

Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@ package loadbalancer
33
import (
44
"errors"
55
"fmt"
6+
"math"
67
"math/rand/v2"
78
"sort"
89
"sync"
@@ -34,6 +35,10 @@ const (
3435

3536
// PowerOfRandomNChoices selects N random endpoints and picks the one with least outstanding requests from them.
3637
PowerOfRandomNChoices
38+
39+
// WeightedRoundRobin distributes requests proportionally to the dynamic
40+
// endpoint weights derived from observed failed round trips.
41+
WeightedRoundRobin
3742
)
3843

3944
const powerOfRandomNChoicesDefaultN = 2
@@ -48,6 +53,7 @@ var (
4853
Random: newRandom,
4954
ConsistentHash: newConsistentHash,
5055
PowerOfRandomNChoices: newPowerOfRandomNChoices,
56+
WeightedRoundRobin: newWeightedRoundRobin,
5157
}
5258
defaultAlgorithm = newRoundRobin
5359
)
@@ -264,6 +270,60 @@ func (p *powerOfRandomNChoices) getScore(e routing.LBEndpoint) int64 {
264270
return -int64(e.Metrics.InflightRequests())
265271
}
266272

273+
type weightedRoundRobin struct {
274+
mu sync.Mutex
275+
rnd *rand.Rand
276+
currentWeights map[string]float64
277+
}
278+
279+
// newWeightedRoundRobin creates a smooth weighted roundrobin algorithm
280+
// with weights based on the failed round trips observed by the endpoint
281+
// registry within the last stats reset period.
282+
func newWeightedRoundRobin(endpoints []string) routing.LBAlgorithm {
283+
return &weightedRoundRobin{
284+
rnd: rand.New(rand.NewPCG(uint64(time.Now().UnixNano()), 0)), // #nosec
285+
currentWeights: make(map[string]float64, len(endpoints)),
286+
}
287+
}
288+
289+
// Apply implements routing.LBAlgorithm with a smooth weighted roundrobin
290+
// algorithm, see https://github.com/nginx/nginx/commit/52327e0627f49dbda1e8db695e63a4b0af4448b1.
291+
// Each endpoint accumulates its weight per round and the endpoint with the
292+
// highest accumulated weight is selected and reduced by the total weight of
293+
// the round. With equal weights it behaves like the roundrobin algorithm.
294+
// Iteration starts at a random offset so that ties are not biased towards
295+
// the first endpoint of the list.
296+
func (w *weightedRoundRobin) Apply(ctx *routing.LBContext) routing.LBEndpoint {
297+
ne := len(ctx.LBEndpoints)
298+
if ne == 1 {
299+
return ctx.LBEndpoints[0]
300+
}
301+
302+
w.mu.Lock()
303+
defer w.mu.Unlock()
304+
305+
total := 0.0
306+
best := 0
307+
bestWeight := math.Inf(-1)
308+
offset := w.rnd.IntN(ne) // #nosec
309+
for k := 0; k < ne; k++ {
310+
i := (offset + k) % ne
311+
e := &ctx.LBEndpoints[i]
312+
weight := e.Metrics.Weight()
313+
total += weight
314+
315+
cw := w.currentWeights[e.Host] + weight
316+
w.currentWeights[e.Host] = cw
317+
if cw > bestWeight {
318+
bestWeight = cw
319+
best = i
320+
}
321+
}
322+
w.currentWeights[ctx.LBEndpoints[best].Host] -= total
323+
324+
return ctx.LBEndpoints[best]
325+
}
326+
267327
type (
268328
algorithmProvider struct{}
269329
initializeAlgorithm func(endpoints []string) routing.LBAlgorithm
@@ -290,6 +350,8 @@ func AlgorithmFromString(a string) (Algorithm, error) {
290350
return ConsistentHash, nil
291351
case "powerOfRandomNChoices":
292352
return PowerOfRandomNChoices, nil
353+
case "weightedRoundRobin":
354+
return WeightedRoundRobin, nil
293355
default:
294356
return None, errors.New("unsupported algorithm")
295357
}
@@ -306,6 +368,8 @@ func (a Algorithm) String() string {
306368
return "consistentHash"
307369
case PowerOfRandomNChoices:
308370
return "powerOfRandomNChoices"
371+
case WeightedRoundRobin:
372+
return "weightedRoundRobin"
309373
default:
310374
return ""
311375
}

loadbalancer/algorithm_test.go

Lines changed: 128 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@ import (
77
"testing"
88

99
"github.com/stretchr/testify/assert"
10+
"github.com/stretchr/testify/require"
1011
"github.com/zalando/skipper/eskip"
1112
"github.com/zalando/skipper/net"
1213
"github.com/zalando/skipper/routing"
@@ -282,6 +283,11 @@ func TestApply(t *testing.T) {
282283
expected: N,
283284
algorithm: newPowerOfRandomNChoices(eps),
284285
algorithmName: "powerOfRandomNChoices",
286+
}, {
287+
name: "weightedRoundRobin algorithm",
288+
expected: N,
289+
algorithm: newWeightedRoundRobin(eps),
290+
algorithmName: "weightedRoundRobin",
285291
}} {
286292
t.Run(tt.name, func(t *testing.T) {
287293
req, _ := http.NewRequest("GET", "http://127.0.0.1:1234/foo", nil)
@@ -573,3 +579,125 @@ func BenchmarkRandomAlgorithm(b *testing.B) {
573579
alg.Apply(lbc)
574580
}
575581
}
582+
583+
type fixedWeightMetrics struct {
584+
routing.Metrics
585+
weight float64
586+
}
587+
588+
func (m fixedWeightMetrics) Weight() float64 { return m.weight }
589+
590+
func setupWeightedRoundRobinRoute(t *testing.T, registry *routing.EndpointRegistry, endpointAddresses []string) *routing.Route {
591+
t.Helper()
592+
provider := NewAlgorithmProvider()
593+
route := &routing.Route{
594+
Route: eskip.Route{
595+
BackendType: eskip.LBBackend,
596+
LBAlgorithm: "weightedRoundRobin",
597+
LBEndpoints: eskip.NewLBEndpoints(endpointAddresses),
598+
},
599+
}
600+
processedRoutes := provider.Do([]*routing.Route{route})
601+
registry.Do([]*routing.Route{route})
602+
require.Len(t, processedRoutes, 1)
603+
return processedRoutes[0]
604+
}
605+
606+
func applyAndCountSelections(t *testing.T, route *routing.Route, rounds int) map[string]int {
607+
t.Helper()
608+
request, err := http.NewRequest("GET", "http://127.0.0.1:1234/foo", nil)
609+
require.NoError(t, err)
610+
lbContext := &routing.LBContext{
611+
Request: request,
612+
Route: route,
613+
LBEndpoints: route.LBEndpoints,
614+
}
615+
616+
selectionCounts := make(map[string]int)
617+
for i := 0; i < rounds; i++ {
618+
selectionCounts[route.LBAlgorithm.Apply(lbContext).Host]++
619+
}
620+
return selectionCounts
621+
}
622+
623+
func TestWeightedRoundRobinDistribution(t *testing.T) {
624+
endpointAddresses := []string{"http://127.0.0.1:1231/foo", "http://127.0.0.1:1232/foo", "http://127.0.0.1:1233/foo"}
625+
endpointWeights := []float64{0.2, 0.8, 1.0}
626+
627+
registry := routing.NewEndpointRegistry(routing.RegistryOptions{})
628+
defer registry.Close()
629+
route := setupWeightedRoundRobinRoute(t, registry, endpointAddresses)
630+
for i := range route.LBEndpoints {
631+
route.LBEndpoints[i].Metrics = fixedWeightMetrics{Metrics: route.LBEndpoints[i].Metrics, weight: endpointWeights[i]}
632+
}
633+
634+
const rounds = 1000
635+
selectionCounts := applyAndCountSelections(t, route, rounds)
636+
637+
// smooth weighted roundrobin distributes proportionally to the weights
638+
totalWeight := 0.0
639+
for _, weight := range endpointWeights {
640+
totalWeight += weight
641+
}
642+
for i, weight := range endpointWeights {
643+
expectedSelections := rounds * weight / totalWeight
644+
assert.InDelta(t, expectedSelections, selectionCounts[route.LBEndpoints[i].Host], 1.0, "endpoint %d", i)
645+
}
646+
}
647+
648+
func TestWeightedRoundRobinEqualWeights(t *testing.T) {
649+
const numberOfEndpoints = 5
650+
endpointAddresses := make([]string, 0, numberOfEndpoints)
651+
for i := 0; i < numberOfEndpoints; i++ {
652+
endpointAddresses = append(endpointAddresses, fmt.Sprintf("http://127.0.0.1:123%d/foo", i))
653+
}
654+
655+
registry := routing.NewEndpointRegistry(routing.RegistryOptions{})
656+
defer registry.Close()
657+
route := setupWeightedRoundRobinRoute(t, registry, endpointAddresses)
658+
659+
// with the default weight of 1.0 every endpoint receives an equal share
660+
const rounds = 1000
661+
selectionCounts := applyAndCountSelections(t, route, rounds)
662+
663+
assert.Len(t, selectionCounts, numberOfEndpoints)
664+
for host, count := range selectionCounts {
665+
assert.Equal(t, rounds/numberOfEndpoints, count, "host %s", host)
666+
}
667+
}
668+
669+
func BenchmarkWeightedRoundRobinAlgorithm(b *testing.B) {
670+
for _, numberOfEndpoints := range []int{10, 100, 1000, 10000} {
671+
b.Run(fmt.Sprintf("%d_endpoints", numberOfEndpoints), func(b *testing.B) {
672+
endpointAddresses := make([]string, numberOfEndpoints)
673+
for i := range numberOfEndpoints {
674+
endpointAddresses[i] = fmt.Sprintf("10.0.%d.%d:8080", i/256, i%256)
675+
}
676+
677+
registry := routing.NewEndpointRegistry(routing.RegistryOptions{})
678+
defer registry.Close()
679+
680+
algorithm := newWeightedRoundRobin(endpointAddresses)
681+
682+
endpoints := make([]routing.LBEndpoint, len(endpointAddresses))
683+
for i := range len(endpointAddresses) {
684+
endpoints[i] = routing.LBEndpoint{
685+
Scheme: "http",
686+
Host: endpointAddresses[i],
687+
Metrics: registry.GetMetrics(endpointAddresses[i]),
688+
}
689+
}
690+
691+
lbContext := &routing.LBContext{
692+
Route: &routing.Route{},
693+
LBEndpoints: endpoints,
694+
}
695+
696+
b.ResetTimer()
697+
698+
for i := 0; i < b.N; i++ {
699+
algorithm.Apply(lbContext)
700+
}
701+
})
702+
}
703+
}

loadbalancer/doc.go

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,16 @@ powerOfRandomNChoices Algorithm
2525
and picks the one with least outstanding requests from them.
2626
Currently, N is 2.
2727
28+
weightedRoundRobin Algorithm
29+
30+
The weightedRoundRobin algorithm distributes requests across the
31+
backend endpoints proportionally to their dynamic weights using
32+
smooth weighted round robin. The weight of an endpoint is the
33+
ratio of its successful round trips within the last stats reset
34+
period of the endpoint registry and requires the passive health
35+
check to be enabled to be updated. With equal weights it behaves
36+
like the roundRobin algorithm.
37+
2838
The load balancing algorithms also provide fade-in behavior for LB endpoints of routes where the
2939
fade-in duration was configured. This feature can be used to gradually add traffic to new instances of
3040
applications that require a certain amount of warm-up time.
@@ -35,6 +45,7 @@ Eskip example:
3545
r2: * -> <consistentHash, "http://127.0.0.1:9998", "http://127.0.0.1:9997">;
3646
r3: * -> <random, "http://127.0.0.1:9998", "http://127.0.0.1:9997">;
3747
r4: * -> <powerOfRandomNChoices, "http://127.0.0.1:9998", "http://127.0.0.1:9997">;
48+
r5: * -> <weightedRoundRobin, "http://127.0.0.1:9998", "http://127.0.0.1:9997">;
3849
3950
Package loadbalancer also implements health checking of pool members for
4051
a group of routes, if backend calls are reported to the loadbalancer.

0 commit comments

Comments
 (0)