fix(docker): apply crawler_configs to single-URL and streaming requests

[talelboussetta]

Summary

Fixes #2287

crawler_configs (#1852) was only honoured when a non-streaming /crawl carried two or more URLs. With one URL the handler called arun(), which takes a single config, and /crawl/stream (and /crawl with stream: true) never passed the list down. Both answered 200 having run with crawler_config alone.

• A request that carries a list goes to arun_many whatever its URL count; arun_many already matches each URL against the list.
• handle_stream_crawl_request takes crawler_configs and applies it; stream_process passes it. Each entry gets stream=True, since arun_many reads the stream flag from the first config of a list. The list entries also get base_config, as on /crawl.
• Loading the list moves into _load_crawler_configs(), shared by both handlers the way _normalize_and_validate_seeds() is, so both apply the same untrusted boundary and wire the PDF URL validator into every entry.
• With a list applied, the list also decides the crawler (_needs_pdf_crawler()): a list of PDFContentScrapingStrategy entries runs on PDFCrawlerStrategy, and since one crawler serves every URL of a request, a list mixing PDF and browser entries is refused with 400 instead of running its PDF entries on the browser crawler. The non-streaming handler now loads the list before it picks a crawler.

@hafezparast: this reverses what test_single_url_ignores_crawler_configs pinned. I read the rationale there ("arun only takes one config") as the reason for the branch rather than a wish to drop the list, so the test is rewritten to assert the new routing. If ignoring it for one URL was deliberate for another reason, I'd like to know.

Not changed: on the streaming path crawler_config itself still doesn't get base_config (only the list entries do, as on /crawl), and a streaming deep crawl still runs from crawler_config, as arun_many itself does. A per-URL deep_crawl_strategy can't reach either path: it is on the untrusted forbidden list, so such a body is rejected when it is loaded.

List of files changed and why

• deploy/docker/api.py: _load_crawler_configs() and _needs_pdf_crawler(); single-URL routing through arun_many when a list is present; the streaming handler's new parameter.
• deploy/docker/server.py: stream_process forwards crawler_configs.
• deploy/docker/tests/test_crawler_configs_routing.py (new): behavioural tests against the real handlers with a recording crawler: one URL with a list, one URL without, streaming with a list, the endpoint forwarding it, a list of PDF entries running on the PDF crawler, and a mixed list refused on both paths, and base_config filling the list on both paths.
• tests/test_issue_1837_config_list.py: the two source assertions that pinned the old branch are updated to the new one.
• deploy/docker/README.md, docs/md_v2/core/self-hosting.md: a "Per-URL Configs" section for crawler_configs: one URL or many on /crawl and /crawl/stream, no crawler_config fallback (end the list with url_matcher: "*"), base_config applied per entry, and a PDF/non-PDF mix returns 400.

How Has This Been Tested?

pytest deploy/docker/tests/test_crawler_configs_routing.py tests/test_issue_1837_config_list.py
23 passed (7 of the 9 new tests fail on `develop`; the other two are controls)

The offline suites (deploy/docker/tests without the live-server test_1–test_7 scripts, tests/unit, the pool tests) show no new failures against develop.

End to end, on an image built from develop @ 1f68e5b (docker build -t c4ai-dev .), against a local test site, so CRAWL4AI_ALLOW_INTERNAL_URLS=true, before and with this branch's api.py and server.py mounted over /app. The page has two sections, .only-a and .only-b; the list sets css_selector: ".only-a" with url_matcher: "*":

# develop
[configs] /crawl, one URL, per-URL css_selector=.only-a -> HTTP 200: A=True B=True
[configs] /crawl/stream, two URLs, same list -> HTTP 200: A=True B=True; A=True B=True

# this branch
[configs] /crawl, one URL, per-URL css_selector=.only-a -> HTTP 200: A=True B=False
[configs] /crawl/stream, two URLs, same list -> HTTP 200: A=True B=False; A=True B=False

cc @ntohidi

Checklist:

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have added/updated unit tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes

[Copilot]

Copilot review overview

🟡 Changes recommended

The streaming path mishandles per-URL PDF crawler selection and silently ignores per-URL deep-crawl strategies.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 1

Open (1)

• Streaming path selects wrong crawler for per-URL PDF configs · New

What changed in this PR

This PR updates Docker crawl routing so crawler_configs applies to single-URL and streaming requests.

Changes:

• Centralizes config-list loading and validation.
• Routes single-URL config lists through arun_many.
• Forwards config lists through streaming handlers.
• Adds and updates routing regression tests.

File	Description
tests/​test_issue_1837_config_list.py	Updates config-list routing expectations.
deploy/​docker/​tests/​test_crawler_configs_routing.py	Adds routing behavior tests.
deploy/​docker/​server.py	Forwards config lists to streaming requests.
deploy/​docker/​api.py	Loads config lists and routes crawl requests.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[ntohidi]

Thanks @talelboussetta for another careful fix. The before/after run with .only-a / .only-b made the bug and the fix easy to see.

What I verified

• Tests: test_crawler_configs_routing.py and test_issue_1837_config_list.py pass on this branch (21 passed). With develop's api.py and server.py, 6 of the 7 new tests fail, as described.
• No regressions: I ran the offline deploy/docker/tests suite (without the live-server test_1–test_7 scripts) on both branches. There are no new failures. The only difference is the 6 tests this PR fixes.
• One URL through arun_many: handle_crawl_request already wraps the result in a list, so the response for one URL looks the same as before.
• Errors: the new 400 for a mixed list is an HTTPException. It passes through the handler's except HTTPException: raise and does not become a 500.
• SSRF: _load_crawler_configs() sets url_validator = validate_url_destination on every PDF entry, on both paths. test_a_list_of_pdf_entries_runs_on_the_pdf_crawler checks this.
• Deep crawl: untrusted bodies cannot set deep_crawl_strategy (see governor.py), so routing through arun_many cannot drop a deep crawl.
• Shared helper: moving the list loading into _load_crawler_configs(), next to _normalize_and_validate_seeds(), keeps both handlers behind the same boundary.

Behaviour changes (for the release notes)

1. One URL whose list has no matching entry: it now fails with "No matching configuration found". Before, it ran with crawler_config. Multi-URL requests already work this way, so the behaviour is now consistent. When a list is sent, crawler_config is not used as a fallback, so users should end the list with a catch-all entry (url_matcher: "*").
2. A list mixing PDFContentScrapingStrategy and other entries: it is now refused with 400. Before, multi-URL requests ran the PDF entries on the browser crawler, which gave bad results for those URLs. Refusing up front is the right call.

Two small requests before we merge

1. Apply base_config on the streaming path too. handle_stream_crawl_request calls _load_crawler_configs(crawler_configs, stream=True) without base_config, so the list gets different defaults on /crawl/stream than on /crawl. Passing config["crawler"]["base_config"] there (with a test) would make both paths the same.
2. Document crawler_configs. No user-facing doc describes it yet. A short section in the Docker deployment docs would help. It should cover: the list is used for one URL or many, on /crawl and /crawl/stream; crawler_config is not a fallback, so end the list with url_matcher: "*"; and mixing PDF and non-PDF entries returns 400.

Once those are in, we'll approve and merge. Thanks again!

[talelboussetta]

Thanks @ntohidi, both are in. I rebased onto develop first, since #2359 touched the same part of api.py.

1. base_config on the streaming path. handle_stream_crawl_request now passes config["crawler"]["base_config"] to _load_crawler_configs, as handle_crawl_request does. test_base_config_fills_the_list_on_both_paths runs one list through both handlers: an entry that leaves a field unset gets the server default, and one that sets it keeps its own. It fails on the streaming path without the change.
2. Docs. A "Per-URL Configs (crawler_configs)" section after Streaming Results in deploy/docker/README.md and docs/md_v2/core/self-hosting.md. It covers the three points you listed (one URL or many on /crawl and /crawl/stream; crawler_config is not a fallback, so end the list with a catch-all url_matcher: "*"; a PDF/non-PDF mix returns 400), plus that base_config is applied to each entry.

Tests: test_crawler_configs_routing.py and test_issue_1837_config_list.py give 23 passed; 7 of the 9 routing tests fail on develop and the other two are controls. The offline deploy/docker/tests suite shows the same 8 failures on develop and on this branch (symlink, file-mode and live-server tests on Windows), so none are new.

[ntohidi]

Thanks @talelboussetta, both changes look good, and thanks for rebasing past #2359.

What I verified

• base_config on streaming: handle_stream_crawl_request now passes config["crawler"]["base_config"], and test_base_config_fills_the_list_on_both_paths checks both handlers give the list the same defaults.
• Docs: the new "Per-URL Configs" section is accurate. I checked the "entry without a url_matcher matches every URL" claim against CrawlerRunConfig.is_match.
• Merge: the branch merges cleanly with current develop. On the merged result, test_crawler_configs_routing.py, test_issue_1837_config_list.py and test_seed_refusal.py pass (27), and deploy/docker/tests has no new failures.

Approving. Thanks again for the careful work!