Skip to content

Commit b66e467

Browse files
attilaedelenyiattilaedelenyi
committed
Add options to rate limit specific user agents to 1 request per hour - e.g boths like SemrushBot
1 parent 59add06 commit b66e467

7 files changed

Lines changed: 695 additions & 163 deletions

File tree

‎charts/ontoserver/README.md‎

Lines changed: 13 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -706,7 +706,18 @@ The `GatewayClass` is named after `ontoserver.gateway.className` and the `EnvoyP
706706
Three optional traffic policies and a data-plane ServiceMonitor can be independently enabled (all require `ontoserver.gateway.enabled: true` and Envoy Gateway CRDs):
707707

708708
- **ClientTrafficPolicy** (`envoygateway.clientTrafficPolicy.enabled`) — applies an HTTP idle timeout on inbound connections from clients. Optionally enables PROXY protocol (`proxyProtocol.enabled`) for upstream load balancers that send PROXY protocol headers (e.g. AWS NLB), and configures client IP detection via `clientIPDetection.xForwardedFor.numTrustedHops` (set to `0` alongside PROXY protocol to use the peer address rather than XFF headers).
709-
- **BackendTrafficPolicy** (`envoygateway.backendTrafficPolicy.enabled`) — caps the maximum upstream request body size and enforces a local rate limit (requests per time unit).
709+
- **BackendTrafficPolicy** (`envoygateway.backendTrafficPolicy.enabled`) — caps the maximum upstream request body size, enforces a local rate limit (requests per time unit), and optionally throttles specific bot user agents. Set `envoygateway.backendTrafficPolicy.blockedUserAgents` to a list of User-Agent substrings (matched as regular expressions) to rate-limit those clients to 1 request per hour — effectively blocking bots such as SemrushBot or AhrefsBot that generate large volumes of traffic:
710+
711+
```yaml
712+
envoygateway:
713+
backendTrafficPolicy:
714+
enabled: true
715+
blockedUserAgents:
716+
- SemrushBot
717+
- AhrefsBot
718+
```
719+
720+
Matched clients receive `429 Too Many Requests`. The global rate limit still applies to all other traffic.
710721
- **SecurityPolicy** (`envoygateway.securityPolicy.enabled`) — enforces IP-based authorization by denying traffic from a list of CIDRs.
711722
- **Gateway ServiceMonitor** (`envoygateway.gatewayServiceMonitor.enabled`) — creates a Prometheus `ServiceMonitor` targeting the Envoy Gateway data-plane pods (scraping `/stats/prometheus` on the `metrics` port). Set `envoygateway.controlPlaneNamespace` to the namespace where Envoy Gateway is installed (default: `envoy-gateway-system`). Requires Prometheus Operator CRDs.
712723

@@ -1019,6 +1030,7 @@ Requires the [External Secrets Operator](https://external-secrets.io/) installed
10191030
| `envoygateway.backendTrafficPolicy.requestBufferLimit` | Max request body size | `1Gi` |
10201031
| `envoygateway.backendTrafficPolicy.rateLimit.requests` | Rate limit requests per unit | `50` |
10211032
| `envoygateway.backendTrafficPolicy.rateLimit.unit` | Rate limit unit (Second, Minute, Hour) | `Second` |
1033+
| `envoygateway.backendTrafficPolicy.blockedUserAgents` | List of User-Agent patterns (regex) to throttle to 1 req/hour — use to block crawlers and bots (e.g. SemrushBot, AhrefsBot) | `[]` |
10221034
| `envoygateway.securityPolicy.enabled` | Enable Envoy SecurityPolicy (requires ontoserver.gateway.enabled) | `false` |
10231035
| `envoygateway.securityPolicy.defaultAction` | Default authorization action (Allow or Deny) | `Allow` |
10241036
| `envoygateway.securityPolicy.deniedCIDRs` | List of CIDRs to deny | `[]` |

‎charts/ontoserver/examples/aks/single-ro-envoy.yaml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -129,6 +129,9 @@ ontoserver:
129129
# rateLimit:
130130
# requests: 50
131131
# unit: Second
132+
# blockedUserAgents: # throttle crawlers to 1 req/hour
133+
# - SemrushBot
134+
# - AhrefsBot
132135
# securityPolicy:
133136
# enabled: true
134137
# defaultAction: Allow

‎charts/ontoserver/templates/envoy-gateway-policies.yaml‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -38,6 +38,16 @@ spec:
3838
type: Local
3939
local:
4040
rules:
41+
{{- range .Values.envoygateway.backendTrafficPolicy.blockedUserAgents }}
42+
- clientSelectors:
43+
- headers:
44+
- name: User-Agent
45+
value: {{ . }}
46+
type: RegularExpression
47+
limit:
48+
requests: 1
49+
unit: Hour
50+
{{- end }}
4151
- limit:
4252
requests: {{ .Values.envoygateway.backendTrafficPolicy.rateLimit.requests }}
4353
unit: {{ .Values.envoygateway.backendTrafficPolicy.rateLimit.unit }}

‎charts/ontoserver/tests/fixtures/blocked-user-agents-values.yaml‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
ontoserver:
2+
gateway:
3+
enabled: true
4+
envoygateway:
5+
backendTrafficPolicy:
6+
enabled: true
7+
blockedUserAgents:
8+
- SemrushBot
9+
- AhrefsBot

‎charts/ontoserver/tests/optional_features_test.yaml‎

Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -87,6 +87,84 @@ tests:
8787
documentIndex: 0
8888
value: RELEASE-NAME-client-settings
8989

90+
- it: renders BackendTrafficPolicy with single global rate limit rule when no blockedUserAgents
91+
templates:
92+
- templates/envoy-gateway-policies.yaml
93+
set:
94+
ontoserver.gateway.enabled: true
95+
envoygateway.backendTrafficPolicy.enabled: true
96+
asserts:
97+
- isKind:
98+
of: BackendTrafficPolicy
99+
documentIndex: 0
100+
- equal:
101+
path: spec.rateLimit.local.rules
102+
documentIndex: 0
103+
value:
104+
- limit:
105+
requests: 50
106+
unit: Second
107+
108+
- it: renders BackendTrafficPolicy with bot-blocking rule when blockedUserAgents configured
109+
templates:
110+
- templates/envoy-gateway-policies.yaml
111+
set:
112+
ontoserver.gateway.enabled: true
113+
envoygateway.backendTrafficPolicy.enabled: true
114+
envoygateway.backendTrafficPolicy.blockedUserAgents[0]: SemrushBot
115+
asserts:
116+
- isKind:
117+
of: BackendTrafficPolicy
118+
documentIndex: 0
119+
- equal:
120+
path: spec.rateLimit.local.rules
121+
documentIndex: 0
122+
value:
123+
- clientSelectors:
124+
- headers:
125+
- name: User-Agent
126+
value: SemrushBot
127+
type: RegularExpression
128+
limit:
129+
requests: 1
130+
unit: Hour
131+
- limit:
132+
requests: 50
133+
unit: Second
134+
135+
- it: renders BackendTrafficPolicy with multiple bot-blocking rules when multiple blockedUserAgents configured
136+
templates:
137+
- templates/envoy-gateway-policies.yaml
138+
values:
139+
- fixtures/blocked-user-agents-values.yaml
140+
asserts:
141+
- isKind:
142+
of: BackendTrafficPolicy
143+
documentIndex: 0
144+
- equal:
145+
path: spec.rateLimit.local.rules
146+
documentIndex: 0
147+
value:
148+
- clientSelectors:
149+
- headers:
150+
- name: User-Agent
151+
value: SemrushBot
152+
type: RegularExpression
153+
limit:
154+
requests: 1
155+
unit: Hour
156+
- clientSelectors:
157+
- headers:
158+
- name: User-Agent
159+
value: AhrefsBot
160+
type: RegularExpression
161+
limit:
162+
requests: 1
163+
unit: Hour
164+
- limit:
165+
requests: 50
166+
unit: Second
167+
90168
- it: renders no PV by default
91169
templates:
92170
- templates/pv.yaml

0 commit comments

Comments
 (0)