Repository navigation
[Bug]: Docker requests can no longer specify deep crawl params #2308
Description
crawl4ai version
0.9.4
Expected Behavior
When calling the Docker /crawl endpoint I should be able to specify:
- Deep or single-page crawl
filter_chainwith params e.g. URL pattern filtersurl_scorerwith params e.g. keywordsmax_pagesandmax_depth(clamped to a configurable limit to prevent DoS)
Current Behavior
Since 0.9.0 any requests with deep_crawl_strategy are rejected with 400.
This removes a ton of functionality that I'm currently relying on - for example:
- If I want to conditionally run single-page vs deep crawls, I would now need to run two docker containers in parallel, each statically configured to do each type.
- I currently dynamically generate URL path filters based on varying input URLs. This is now impossible because they cannot be defined statically.
According to the changelog, this was removed from the API to improve security. However, I believe this could be re-added with no increased security risk, beyond clamping a few values (e.g. max pages) to a server-configurable limit to prevent DoS - which seems to already be implemented in deploy/docker/governor.py.
An alternative would be to allow users to opt-out of such security limitations, for example in environments where they are already validating inputs before calling the API.
OS
Linux
Python version
3.12
Browser
No response
Browser version
No response
Error logs & Screenshots (if applicable)
No response
Activity
[-][Bug]: Docker requests can not longer specify single or deep crawl, and safe deep crawl params[/-][+][Bug]: Docker requests can no longer specify single or deep crawl, and safe deep crawl params[/+][-][Bug]: Docker requests can no longer specify single or deep crawl, and safe deep crawl params[/-][+][Bug]: Docker requests can no longer specify deep crawl params[/+]SohamKukreti commentedon Oct 5, 2026
Thanks for raising this issue @markwainwright.
I've raised PR #2340 which adds an opt-in parameter
CRAWL4AI_ALLOW_DEEP_CRAWL(off by default) and also patches some security holes in deep crawl.Feel free to try it out and let us know if it works for you!
markwainwright commentedon Oct 8, 2026
@SohamKukreti Yep, I tested #2340 out and it works perfectly for my use case. Thanks for your work on this!
feat(docker): opt-in deep crawling via CRAWL4AI_ALLOW_DEEP_CRAWL (#2308)
ntohidi commentedon Oct 13, 2026
The opt-in parameter is in the
developbranch and will be released soon.ntohidi commentedon Oct 14, 2026
Fixed by #2340, released in v0.9.5. Upgrade with
pip install -U crawl4ai. Release notes: https://github.com/unclecode/crawl4ai/blob/main/docs/blog/release-v0.9.5.md