You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 8ba10a9
Browse filesBrowse the repository at this point in the historyBrowse files
Copy file name to clipboardExpand all lines: app/_ai_gateway_entities/ai-mcp-server.md
+14-28Lines changed: 14 additions & 28 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -549,42 +549,28 @@ config:
549
549
- kongair-internal-root-ca
550
550
```
551
551
552
-
## Rate limiting {% new_in 2.3 %}
552
+
## Rate limiting
553
553
554
-
The AI MCP Server [`rate_limiting`](#schema-aigateway-mcpserver-rate-limiting) field caps how many requests the server accepts for a specific MCP method or [tool](#tools).
555
-
For example, you can allow `tools/list` frequently but restrict an expensive `search_flights` tool to a few calls per second.
554
+
Some MCP operations cost more than others.
555
+
A single `tools/call` to a flight search tool can hit a metered upstream API, while `tools/list` is cheap.
556
+
The AI MCP Server [`rate_limiting`](#schema-aigateway-mcpserver-rate-limiting) field lets you cap requests for a specific MCP method or [tool](#tools), so an expensive tool or a runaway client can't exhaust your upstream quota.
556
557
557
-
The `rate_limiting` field is supported in `conversion-listener` and `passthrough-listener` modes.
558
-
Other [server modes](#server-modes) reject it.
558
+
Use `rate_limiting` in the following cases:
559
559
560
-
Set a list of `limits`.
561
-
Each entry in the list has the following fields:
560
+
- **Protect an expensive tool**: limit a single tool, such as `search_flights`, to a small number of calls per second, without slowing down the rest of the server.
561
+
- **Cap overall tool usage**: limit all `tools/call` requests per minute, regardless of which tool the client calls.
562
+
- **Layer short and long windows**: combine a per-second limit to absorb bursts with a per-day limit to control total usage for the same tool.
562
563
563
-
{% table %}
564
-
columns:
565
-
- title: Field
566
-
key: field
567
-
- title: Description
568
-
key: description
569
-
rows:
570
-
- field: "`method`"
571
-
description: "The MCP method the limit applies to, for example `tools/call`. Set either `method` or `tool`, not both."
572
-
- field: "`tool`"
573
-
description: "The name of the tool the limit applies to, for example `search_flights`. Set either `method` or `tool`, not both."
574
-
- field: "`limit`"
575
-
description: "The number of requests allowed per window. Must be between 1 and 2147483646."
576
-
- field: "`window_size`"
577
-
description: "The window size in seconds. Must be `1` (second), `60` (minute), `3600` (hour), or `86400` (day)."
578
-
{% endtable %}
564
+
The `rate_limiting` field is supported in `conversion-listener` and `passthrough-listener` modes.
565
+
Other [server modes](#server-modes) reject it.
579
566
580
-
Each limit must set exactly one of `method` or `tool`.
581
-
Each combination of target and `window_size` can appear only once.
582
-
You can set several windows for the same target, for example a per-hour and a per-day limit on one tool.
583
-
If either rule is broken, the API returns a `400` error that points to the invalid entry, for example `rate_limiting.limits[1]`.
567
+
Each limit targets either an MCP method or a tool name, never both, and sets a request count for a fixed window of one second, one minute, one hour, or one day.
568
+
You can define several windows for the same target, but each combination of target and window can appear only once.
569
+
If a limit breaks either rule, the API returns a `400` error that points to the invalid entry, for example `rate_limiting.limits[1]`.
584
570
585
571
Each {{site.ai_gateway}} data plane node keeps its own counters and enforces the limits independently.
586
572
587
-
In the following example, the AI MCP Server allows 100 `tools/call` requests per minute.
573
+
In the following example, the AI MCP Server allows 100 `tools/call` requests per minute across all tools.
588
574
It also limits the `search_flights` tool to 10 requests per second and 1000 requests per day:
0 commit comments