Repository navigation
Release v0.9.5 - #2368
Merged
Merged
Release v0.9.5#2368
Conversation
Gate the SSE inbox so initialize is processed before concurrent tools/call POSTs, fixing -32602 rejections on connect (#2233).
Pages that nest <noscript> lose a closing tag when a scripting-enabled browser serializes them, so lxml nests the rest of the document inside the still-open element and the noscript removal deletes the entire page, leaving cleaned_html with an empty body. Unwrap noscript tags (keeping their contents) when opening and closing counts disagree; well-formed documents are untouched. Co-authored-by: Cursor <cursoragent@cursor.com>
CrawlResult.links and .media defaulted to {}. Successful crawls fill them from
the scraper's result, but the results AsyncWebCrawler builds itself -- the
robots.txt refusal, "All proxies failed", the exception path in arun, the
anti-bot raw-HTML fallback -- never set them, so those results had no
"internal"/"external" or "images" keys: result.links["internal"] raised
KeyError, and the Docker API returned "links": {} for exactly the entries of a
batch a client has to handle.
Default both to the keys a scraped page gets -- {"internal": [], "external": []}
and {"images": [], "videos": [], "audios": []} ("tables" is moved out of media
into CrawlResult.tables on the success path) -- so every construction site is
covered, including future ones.
…n test
The CrawlResult sketches in docs/md_v2/api/crawl-result.md,
docs/md_v2/core/crawler-result.md and the /execute_js docstring still showed
`= {}` for links and media. The isolation test now also asserts the nested
lists are distinct objects, and compares against fresh literals.
complete-sdk-reference.md repeats the CrawlResult sketch from
docs/md_v2/api/crawl-result.md and still showed `= {}` for links and media.
#2309) BFS, DFS and best-first strategies reset their cancel event at the start of each run but kept _pages_crawled from the previous run, so reusing one strategy for independent crawls shared a single max_pages budget. Reset the counter in the non-resume branch of each run; a resumed run still restores pages_crawled from its saved state.
CacheValidator built its own httpx client without proxy=proxy_url(), so check_cache_freshness (untrusted-allowlisted) let an API client make the Docker server send HEAD and GET requests to internal addresses, directly or through a redirect. The client now uses the installed egress proxy like the seeder and robots.txt fetches. No change when no proxy is installed.
… from untrusted bodies Untrusted request bodies may not supply page JavaScript (js_code is forbidden), but two allowlisted fields reached the same page.evaluate() sink. _crawl_web pasted each css_selector part into an f-string, so a '"' ended the string literal and the rest ran as script. smart_wait runs a wait_for that starts with "js:", looks like a function, or is not a valid selector as JavaScript. css_selector is now passed to the page as an argument, which also makes selectors with quotes such as p[title="x"] work. For untrusted bodies, JavaScript wait_for values are refused and a bare value is pinned to the "css:" branch, which never falls back to JavaScript.
CosineStrategy has no untrusted field allowlist, so every field that was not explicitly forbidden reached the constructor. model_name goes straight to AutoModel.from_pretrained, which let an API client pick the HuggingFace repository the server downloads and loads; with torch older than 2.6 a pickle weight file in that repository runs code in the server process. Untrusted bodies now always get the default model.
RegexExtractionStrategy.custom, RegexChunking.patterns and schema fields of type "regex" (JsonCss/JsonXPath/JsonLxml) let an API client run its own pattern over page content it also controls. A catastrophic-backtracking pattern runs for hours, and Python's re holds the GIL while it matches, so one request froze the whole server; the crawl deadline cannot interrupt it. Untrusted bodies may no longer supply any of these. The built-in RegexExtractionStrategy patterns stay available.
delay_before_return_html, mean_delay and max_range are on the untrusted allowlist but _clamp_untrusted left them unbounded, so one request could hold a browser page or stall the arun_many rate limiter for as long as it liked. They are now capped at 60 seconds; negative, NaN and non-numeric values fall back to the field default.
_extract_domain took everything between "://" and the next "/", so URL
userinfo, a port or a trailing dot ("https://user@evil.com/",
"https://evil.com:443/", "https://evil.com./") slipped past
blocked_domains. The host now comes from urlparse().hostname with the
trailing dot stripped, and configured domains are normalised the same way.
The company list, org chart, person details, hover info and chat panes built HTML from crawled names, descriptions and URLs with innerHTML and no escaping, so a crafted profile ran script when the graph view was opened. Text now goes through esc(), src/href values only accept http(s) URLs and are escaped for the attribute, and the markdown chat output is passed through DOMPurify.
- Header-name and artifact-id checks used re.match with a "$" anchor, which also accepts a trailing newline. They now use fullmatch. - clamp_deep_crawl logged nothing when it could not set max_pages or max_depth. It now logs a warning. - enforce_egress ignored a failure to pin the browser to the egress proxy and let the crawl run unpinned. It now logs and refuses with EgressBlocked.
The public apps and articles endpoints built WHERE clauses from query and path parameters with f-strings, so an unauthenticated client could read any table through UNION. get_all() now takes bound parameters and every caller passes values that way. Admin create and update handlers accept only columns declared in schema.yaml, and errors return a generic message instead of the raw exception text.
Both default launch flag lists carried --ignore-certificate-errors, which turns off TLS verification for the whole browser, so ignore_https_errors=False (and the Docker server's TLS-on default) had no effect. The flag is gone; insecure mode comes from the context option ignore_https_errors, and only the CDP-managed subprocess, whose default context takes no such option, adds the flag when ignore_https_errors is True. The Docker fallback config drops it and --allow-insecure-localhost too. CRAWL4AI_CHROMIUM_SANDBOX=true now reaches every browser, including ones built from a request body: --no-sandbox is removed from the base flags and extra_args, and Playwright gets chromium_sandbox=True. The posture test now checks the real launch flags instead of config.yml.
…d-shape fix(models): default CrawlResult links and media to the scraped shape
One seed that fails the destination check (internal address, or a host that does not resolve) used to raise 400 for the whole /crawl or /crawl/stream request. Now the refused seed comes back as a failed CrawlResult with the same opaque message, and the other seeds are crawled. If every seed is refused, the request is still a 400. Fixes #2288
Siblings in a level queue links before the level's own pages are all counted, so the next level could still exceed max_pages in a real batch crawl (83 pages for max_pages=50). Cut each level to the budget left before fetching it, in both batch and stream mode. Add tests that run the real batch and stream loops against a fake crawler.
fix(docker): report a refused seed as a failed result, not a dead batch
…pties-cleaned-html Fix: keep page content nested in an unclosed <noscript>
The per-URL config list (#1837) was only honoured when a non-streaming request carried two or more URLs. With one URL the handler called arun(), which takes a single config, and /crawl/stream (and /crawl with stream=true) never passed the list to handle_stream_crawl_request. Both requests ran with crawler_config alone and still answered 200. A request that carries a list now goes to arun_many whatever its URL count, and the streaming handler takes and applies the list. Loading moves into _load_crawler_configs(), shared by both handlers the way _normalize_and_validate_seeds() is, so both apply the same trust boundary and wire the PDF URL validator into every entry. On the streaming path each entry gets stream=True, since arun_many reads the stream flag from the first config of a list. The legacy source-grep test that pinned the single-URL path to arun() is updated to the new routing; behavioural tests are in deploy/docker/tests/test_crawler_configs_routing.py.
With crawler_configs applied, the crawler was still chosen from the top-level crawler_config alone, so a list entry using PDFContentScrapingStrategy ran on the pooled browser crawler and failed before the PDF scraper could run. _needs_pdf_crawler() decides from the list when there is one: a list of PDF entries runs on PDFCrawlerStrategy, and since one crawler serves every URL of a request, a list mixing PDF and browser entries is refused with 400 instead of failing mid-crawl. The non-streaming handler now loads the list before it picks a crawler, as the streaming one already did.
Wraps the new lines over black's limit. The source assertion in test_issue_1837_config_list.py now matches the call rather than the whole one-line conditional, so it no longer depends on how the line is wrapped. No behaviour change.
preprocess_html_for_schema parses the raw HTML separately from the scraper and removes <noscript>, so pages with nested noscript still lost their body in fit_html after #2284. Reuse unwrap_orphaned_noscript() before parsing.
fix(utils): keep fit_html body nested in an unclosed <noscript>
fix(deep-crawl): reset page budget on a fresh run of a reused strateg…
Pass the page number explicitly to _process_page() and _extract_images() instead of sharing it through self.current_page_number, which worker threads in process_batch() raced on. The race misapplied page-1-only formatting, corrupted PDFPage.page_number, and could collide extracted image filenames across pages. Add regression tests covering batch determinism, page-1-only formatting, sequential page numbering, and serial/batch parity. Signed-off-by: Kylinny <j.yao@wustl.edu>
Errors raised from AsyncHTTPCrawlerStrategy._handle_http quoted the underlying aiohttp message, and InvalidURL's message is the whole proxy URL with the password in it. Mask user:pass@ in the wrapped message, and when the original carried credentials keep it out of the traceback. Refs #2326
Show a clear usage error instead of crashing on None, remove the duplicate JSON print, and fix the -o json example and profile menu choice.
…ntials-encoding Percent-encode proxy credentials in the HTTP crawler proxy URL
…tials-in-http-errors fix(http): keep proxy credentials out of HTTP crawler errors
…unting tags unwrap_orphaned_noscript counted <noscript> tags with a regex over the raw HTML, so a "<noscript>" string inside a <script>, <style> or HTML comment made a well-formed page look unbalanced. Every noscript was then unwrapped and its fallback text leaked into cleaned_html. Count on a copy with those raw-text regions removed; unwrap the original as before. Follow-up to #2284 (issue #2293).
remove_empty_elements_fast dropped whitespace-only childless elements such
as <span> </span>, so words split across inline elements were joined in
the markdown ("Hi there" became "Hithere"). Merge a single space into
the previous sibling's tail or the parent's text in place of the element.
Fixes #2328
fix(scraping): keep a space where a whitespace-only element is removed
…ments A <script>, <style> or <!-- with no closing tag made the lazy match scan to the end of the document from every such opener, so a 1 MB page with 20k unclosed openers took over a minute. Let an unclosed one run to the end of the document, as a browser does, so the scan is linear.
fix: ignore <noscript> text inside script, style and comments when counting tags
Fail early when `-o json` has no extraction strategy
docs(cli): clarify that --output json requires extraction strategy
fix(deep-crawl): count queued next-level URLs against max_pages in BFS
Requests may set deep_crawl_strategy when the flag is true, with safe types only, glob-only URL patterns, config page/depth limits and a deadline on streamed deep crawls.
Refuse nested CompositeScorer, limit one strategy to 20 filters and scorers and 200 list or dict items, and treat limits.max_pages: 0 as no limit.x
Opt-in deep crawling via `CRAWL4AI_ALLOW_DEEP_CRAWL` parameter
MCP: Order SSE handshake before early tool calls
…2365) --bypass-cache was a flag with default=True, so it always forced CacheMode.BYPASS after -c and config files were applied. Default it to False; CrawlerRunConfig already defaults to BYPASS, so plain crwl still skips the cache. -c and YAML/JSON configs also pass cache_mode as a string, which never matched the CacheMode enum, so CrawlerRunConfig now converts it.
CrawlerRunConfig now converts a string cache_mode to CacheMode and raises ValueError on an unknown one, which the Docker API turned into a 500. Check it in the untrusted gate so /crawl and /crawl/stream answer 400. The playground template test compared the raw "bypass" string with the loaded config; cache_mode is now a CacheMode, so compare enum values.
fix(cli): honor -c cache_mode instead of always bypassing the cache (#2365)
# Conflicts: # crawl4ai/async_configs.py # deploy/docker/governor.py
Bump the version to 0.9.5.
Rename .env.txt to .env.example, move JOURNAL.md to docs/deprecated/, and remove two stale webhook check scripts from the root. Webhook coverage lives in deploy/docker/tests/.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
10 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Release v0.9.5. Already tagged and published to PyPI from
838869b. This PR bringsmainup to the release.css_selectorandwait_forcan no longer run page JavaScript,CosineStrategy.model_nameand caller-supplied regexes are refused, delay fields are capped, and Chromium no longer launches with--ignore-certificate-errors.CRAWL4AI_ALLOW_DEEP_CRAWL=true.<noscript>pages, proxy credentials, the CLI (-c cache_mode,-o json),crawler_configs, and MCP over SSE..env.txtrenamed to.env.example,JOURNAL.mdmoved todocs/deprecated/, two stale webhook scripts removed.No breaking changes for library users. On the Docker server, untrusted bodies that set
CosineStrategy.model_name, custom regexes, or a JavaScriptwait_fornow get a 400.Release notes: docs/blog/release-v0.9.5.md
origin/mainis an ancestor of this branch, so it merges as a fast-forward.