Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
65 commits
Select commit Hold shift + click to select a range
d5d26a7
fix(mcp): order SSE handshake before early tool calls
SohamKukreti Sep 16, 2026
6b5a0d4
Fix: keep page content nested in an unclosed <noscript>
Sep 24, 2026
8bd0776
fix(models): default CrawlResult links and media to the scraped shape
talelboussetta Sep 25, 2026
1148176
docs(models): show the new CrawlResult defaults; tighten the isolatio…
talelboussetta Sep 25, 2026
e88c441
docs(models): show the new CrawlResult defaults in the SDK reference
talelboussetta Sep 26, 2026
c24d12d
fix(deep-crawl): reset page budget on a fresh run of a reused strateg…
Artechie0806 Sep 29, 2026
4acca11
fix(security): route the cache freshness check through the egress proxy
ntohidi Sep 29, 2026
736a28a
fix(security): stop css_selector and wait_for running page JavaScript…
ntohidi Oct 2, 2026
9bfe9c5
fix(security): refuse CosineStrategy.model_name from untrusted bodies
ntohidi Oct 2, 2026
8aea200
fix(security): refuse caller-supplied regexes from untrusted bodies
ntohidi Oct 2, 2026
d6fdff5
fix(security): cap delay fields in untrusted CrawlerRunConfig bodies
ntohidi Oct 5, 2026
6557e96
fix(security): parse the real host in DomainFilter
ntohidi Oct 5, 2026
5fff5b9
fix(security): escape crawled data in the LinkedIn graph view template
ntohidi Oct 5, 2026
171453f
fix(security): tighten Docker hardening guards
ntohidi Oct 5, 2026
978dbb5
fix(security): parameterise marketplace backend queries
ntohidi Oct 5, 2026
f0ada4c
fix(security): stop launching Chromium with certificate checks off
ntohidi Oct 5, 2026
9abbb47
docs: credit Chunyi Liu and Julian Godfrey in SECURITY-CREDITS.md
ntohidi Oct 5, 2026
7e3518e
Merge pull request #2289 from talelboussetta/bugfix/crawlresult-faile…
ntohidi Oct 12, 2026
350a141
fix(docker): report a refused seed as a failed result, not a dead batch
ntohidi Oct 12, 2026
860072e
fix(deep-crawl): count queued next-level URLs against max_pages in BF…
drakeo338 Oct 5, 2026
7e76f20
fix(deep-crawl): cap each BFS level to the remaining max_pages budget
drakeo338 Oct 12, 2026
3e12a96
Merge pull request #2359 from unclecode/fix/2288-per-seed-refusal
ntohidi Oct 12, 2026
07ff307
Merge pull request #2284 from tommypacker/bugfix/unclosed-noscript-em…
ntohidi Oct 12, 2026
81efcc3
fix(docker): apply crawler_configs to single-URL and streaming requests
talelboussetta Sep 25, 2026
5ce402f
fix(docker): let a per-URL config list decide the PDF crawler
talelboussetta Sep 25, 2026
4d645a4
style(docker): black-format the new crawler_configs code
talelboussetta Sep 26, 2026
ba8b722
fix(utils): keep fit_html body nested in an unclosed <noscript>
ntohidi Oct 12, 2026
89c2642
Merge pull request #2360 from unclecode/fix/2293-fit-html-noscript
ntohidi Oct 12, 2026
82af20f
Merge pull request #2317 from Artechie0806/fix/2309-reset-pages-crawled
ntohidi Oct 12, 2026
46c7905
Fix nondeterministic PDF markdown with batch_size > 1 (#2319)
Kylinny Sep 30, 2026
3bd2450
Merge pull request #2320 from Kylinny/fix-pdf-page-number-race-2319
ntohidi Oct 12, 2026
974275d
fix(docker): apply base_config to the crawler_configs list on the str…
talelboussetta Oct 12, 2026
8276964
docs(docker): document crawler_configs
talelboussetta Oct 12, 2026
36cea05
docs(cli): clarify that --output json requires an extraction strategy…
ege-arhan Oct 4, 2026
2229adc
docs(cli): update -o json help text per maintainer review
ege-arhan Oct 12, 2026
4268340
Merge pull request #2290 from talelboussetta/bugfix/crawler-configs-s…
ntohidi Oct 12, 2026
262271d
fix(http): keep proxy credentials out of HTTP crawler errors
alkaz-nodemaven Oct 12, 2026
2de19c8
Percent-encode proxy credentials in the HTTP crawler proxy URL
omarbounawarapy Oct 1, 2026
2432985
Format new proxy URL test with black
omarbounawarapy Oct 1, 2026
8b3a75d
Decode percent-encoded credentials in ProxyConfig.from_string
omarbounawarapy Oct 12, 2026
1e54fa7
fix(cli): fail early when -o json has no extraction strategy
SohamKukreti Oct 12, 2026
bcad146
Merge pull request #2327 from omarbounawarapy/bugfix/http-proxy-crede…
ntohidi Oct 12, 2026
01d5565
Merge pull request #2361 from alkaz-nodemaven/fix/redact-proxy-creden…
ntohidi Oct 12, 2026
b59dc9c
fix: ignore <noscript> text inside script, style and comments when co…
ntohidi Oct 12, 2026
c381413
fix(scraping): keep a space where a whitespace-only element is removed
drakeo338 Oct 2, 2026
5bc4e19
Merge pull request #2329 from drakeo338/claude/2328-fix
ntohidi Oct 12, 2026
3442707
fix: keep the raw-text strip linear on unclosed script, style and com…
ntohidi Oct 12, 2026
a6e3bb3
Merge pull request #2363 from unclecode/fix/noscript-count-skip-raw-text
ntohidi Oct 12, 2026
0335a2e
Merge pull request #2362 from unclecode/fix/cli-json-output-2334
ntohidi Oct 12, 2026
29c28eb
Merge pull request #2335 from ege-arhan/fix/cli-json-output-help
ntohidi Oct 12, 2026
1fb35bf
Merge pull request #2339 from drakeo338/claude/2338-fix
ntohidi Oct 12, 2026
a4312ac
feat(docker): opt-in deep crawling via CRAWL4AI_ALLOW_DEEP_CRAWL (#2308)
SohamKukreti Oct 5, 2026
d6f8a75
feat(docker): total size budget for untrusted deep-crawl strategies
SohamKukreti Oct 13, 2026
186bddb
Merge pull request #2340 from unclecode/feat/docker-deep-crawl-opt-in
ntohidi Oct 13, 2026
99679fc
Merge pull request #2273 from unclecode/fix/2233-mcp-sse-init-gate
ntohidi Oct 13, 2026
018053b
fix(cli): honor -c cache_mode instead of always bypassing the cache (…
ntohidi Oct 13, 2026
ae4a9b6
fix(api): refuse an invalid cache_mode string with 400, not 500
ntohidi Oct 14, 2026
f475c3b
Merge pull request #2366 from unclecode/fix/2365-cli-cache-mode
ntohidi Oct 14, 2026
2921d4b
Merge security fixes for 0.9.5
ntohidi Oct 14, 2026
ddb6ac1
test: let the clamp-failure stub accept filter_chain after the merge
ntohidi Oct 14, 2026
b834641
docs: release notes, changelog, README, and security notes for v0.9.5
ntohidi Oct 14, 2026
31ab16d
docs: add the GHSA ids for the v0.9.5 advisories
ntohidi Oct 14, 2026
eef62a4
docs: add Chunyi Liu's GitHub handle to the v0.9.5 credits
ntohidi Oct 14, 2026
5b8fd5b
chore: tidy the repository root
ntohidi Oct 14, 2026
838869b
Merge main into develop: cloud launch wording
ntohidi Oct 14, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
File renamed without changes.
64 changes: 64 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,70 @@ All notable changes to Crawl4AI will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [0.9.5] - 2026-10-14

0.9.5 is a security release. It closes six coordinated-disclosure advisories in the Docker server's handling of untrusted request bodies: one more SSRF path outside the egress proxy, page JavaScript injection through `css_selector` and `wait_for`, a caller-chosen HuggingFace model in `CosineStrategy`, ReDoS through caller-supplied regexes, unbounded delay fields, and Chromium launched with certificate checks off. It also adds opt-in deep crawling on the Docker server and ships the bug fixes that landed on `develop` since 0.9.4. Users who self-host the Docker server should upgrade.

### Security

- **Blind SSRF via the cache freshness check (CWE-918, medium)**: `CacheValidator.validate()` sent its HEAD and GET requests on its own HTTP client, outside the Docker server's pinning egress proxy. It now goes through the proxy like the seeder and robots.txt fetches fixed in 0.9.4. Credit: [arpe1618](https://github.com/arpe1618). (GHSA-w4xc-px5p-c32w)
- **Page JavaScript injection via `css_selector` and `wait_for` (CWE-95, medium)**: `css_selector` was pasted into a JavaScript string, and `wait_for` could run as page script, so an untrusted body could run JavaScript in the crawl page even though `js_code` is refused. `css_selector` is now passed to `evaluate()` as an argument, which also makes selectors with quotes work. For untrusted bodies, JavaScript `wait_for` values are refused and a bare value is treated as a CSS selector. Credit: Zhixi "Jace" Sun ([manus-use](https://github.com/manus-use)). (GHSA-qj46-wp7m-rgj8)
- **Caller-chosen model in `CosineStrategy` (CWE-502, high)**: an untrusted body could set `CosineStrategy.model_name`, so the server downloaded and loaded any HuggingFace repository. With PyTorch older than 2.6 a crafted pickle weight file runs code in the server. `model_name` is now refused from untrusted bodies. The default image does not install PyTorch and is not affected. Credit: Zhixi "Jace" Sun ([manus-use](https://github.com/manus-use)). (GHSA-f856-7xp9-w3cc)
- **ReDoS via caller-supplied regexes (CWE-1333, medium)**: `RegexExtractionStrategy.custom`, `RegexChunking.patterns`, and `"type": "regex"` fields in extraction schemas let a client send a catastrophic-backtracking pattern. Python's `re` holds the GIL while it matches, so one request froze the whole server. These fields are now refused from untrusted bodies. The built-in patterns stay available. Credit: Zhixi "Jace" Sun ([manus-use](https://github.com/manus-use)). (GHSA-j843-8p47-rhxp)
- **Unbounded delay fields (CWE-770, medium)**: `delay_before_return_html`, `mean_delay`, and `max_range` were allowed in untrusted bodies with no upper bound, so one request could hold a browser page or stall the rate limiter. They are now capped at 60 seconds, and negative, NaN, and non-numeric values fall back to the default. Credit: Chunyi Liu ([liuchunyi-buaa](https://github.com/liuchunyi-buaa)). (GHSA-rhjm-7hj7-75fx)
- **Chromium launched with certificate checks off (CWE-295, medium)**: both default launch flag lists carried `--ignore-certificate-errors`, so `ignore_https_errors=False` and the Docker server's TLS-on default had no effect. The flag is gone; insecure mode now comes only from `ignore_https_errors`. `CRAWL4AI_CHROMIUM_SANDBOX=true` now reaches every browser, including ones built from a request body. Credit: Julian Godfrey. (GHSA-j2qh-f73r-mg3q)

Hardening, no advisory (credit: Chunyi Liu, [liuchunyi-buaa](https://github.com/liuchunyi-buaa)):

- `DomainFilter` parses the real host with `urlparse()`, so userinfo, a port, or a trailing dot no longer slips past `blocked_domains`.
- Webhook header names and artifact ids are checked with `fullmatch`, so a trailing newline no longer passes.
- `enforce_egress` refuses the crawl with `EgressBlocked` when it cannot pin the browser to the egress proxy, instead of running unpinned. `clamp_deep_crawl` logs a warning when it cannot clamp.
- Documentation examples: the marketplace backend (`docs/md_v2/marketplace/backend/`) uses bound SQL parameters and returns a generic error, and the LinkedIn graph view example escapes crawled data.

All reporters are credited in `SECURITY-CREDITS.md`. GitHub Security Advisories accompany this release.

### Added

- Docker server: opt-in deep crawling. With `CRAWL4AI_ALLOW_DEEP_CRAWL=true`, requests may set `deep_crawl_strategy` with BFS, DFS, or BestFirst, safe filter and scorer types, and glob-only URL patterns. Page and depth budgets come from `limits.max_pages` and `limits.max_depth` in `config.yml`, a strategy is limited to 20 filters and scorers, and streamed deep crawls get a deadline. Off by default. (#2340, issue #2308, thanks @SohamKukreti)
- A short Crawl4AI Cloud notice at `crawl4ai-setup`, at first run, and when a crawl is blocked. `CRAWL4AI_NO_CLOUD_NOTICE=1` turns it off.

### Fixed

**Crawler and core**

- Deep crawl: BFS counts URLs already queued for the next level against `max_pages`, so `max_pages=50` no longer fetches hundreds of pages. (#2339, issue #2338, thanks @drakeo338)
- Deep crawl: a reused strategy resets its page budget on a fresh run. (#2317, issue #2309, thanks @Artechie0806)
- Scraping: page content nested in an unclosed `<noscript>` is kept in `cleaned_html` and `fit_html`. (#2284, thanks @tommypacker; #2360, #2363, issue #2293)
- Scraping: a space is kept where a whitespace-only element is removed, so words no longer run together. (#2329, issue #2328, thanks @drakeo338)
- `CrawlResult.links` and `.media` default to the scraped shape (`internal`, `external`, `images`, ...) on failed results. (#2289, issue #2286, thanks @talelboussetta)
- PDF: markdown is deterministic with `batch_size > 1`. (#2320, issue #2319, thanks @Kylinny)
- HTTP crawler: proxy credentials with `#`, `/`, `?`, or `%` work, and `ProxyConfig.from_string` decodes percent-encoded credentials. (#2327, issue #2326, thanks @omarbounawarapy)
- HTTP crawler: proxy credentials no longer appear in error messages. (#2361, thanks @alkaz-nodemaven)

**CLI**

- `crwl -c cache_mode=...` is honored instead of always bypassing the cache. (#2366, issue #2365)
- `crwl -o json` without an extraction strategy fails early with a clear message instead of crashing after the crawl. (#2362, #2335, issue #2334, thanks @SohamKukreti and @ege-arhan)

**Docker server**

- `crawler_configs` applies to single-URL and streaming requests, and a per-URL config list decides the PDF crawler. (#2290, issue #2287, thanks @talelboussetta)
- A seed that fails the destination check is returned as a failed result instead of failing the whole batch with 400. (#2359, issue #2288)
- An invalid `cache_mode` string returns 400 instead of 500.
- MCP over SSE: `initialize` is processed before early `tools/call` requests, so fan-out clients no longer get -32602 on connect. (#2273, issue #2233, thanks @SohamKukreti)

### Tests

- `tests/unit/test_egress_policy.py` and `deploy/docker/tests/test_security_ssrf_seeder.py`: the cache freshness check goes through the egress proxy.
- `tests/unit/test_untrusted_field_sinks.py` and `tests/browser/test_css_selector_injection.py`: `css_selector`, `wait_for`, `model_name`, regex, and delay fields under the untrusted gate.
- `tests/unit/test_browser_tls_sandbox_flags.py`: launch flags with and without `ignore_https_errors` and the sandbox opt-in.
- `tests/unit/test_domain_filter_host.py`, `deploy/docker/tests/test_security_hardening_2026_10.py`, `docs/md_v2/marketplace/backend/test_sql_injection.py`: the hardening fixes.
- Coverage for the deep-crawl, noscript, proxy, CLI, and Docker fixes.

### Breaking Changes

None for library users. On the Docker server, untrusted request bodies that set `CosineStrategy.model_name`, `RegexExtractionStrategy.custom`, `RegexChunking.patterns`, a `"type": "regex"` schema field, or a JavaScript `wait_for` are now refused with 400.

## [0.9.4] - 2026-09-23

0.9.4 is a security release. It closes three coordinated-disclosure advisories: two SSRF paths that bypassed the Docker server's egress controls, and a trust-boundary bypass that let a non-admin API client read server environment variables. It also makes content pruning about 10x faster with the new lxml-native `PruningContentFilterLXML`, now the default, and ships the bug fixes that accumulated on `develop` since 0.9.3. There are no breaking changes. Users who self-host the Docker server should upgrade.
Expand Down
2 changes: 1 addition & 1 deletion Dockerfile
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
FROM python:3.12-slim-bookworm AS build

# C4ai version
ARG C4AI_VER=0.9.4
ARG C4AI_VER=0.9.5
ENV C4AI_VERSION=$C4AI_VER
LABEL c4ai.version=$C4AI_VER

Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@
[![Discord](https://img.shields.io/badge/Discord-join%20us-5865F2?logo=discord&logoColor=white)](https://discord.gg/jP8KfhDhyN)
[![Crawl4AI Cloud](https://img.shields.io/badge/Crawl4AI_Cloud-try_it_free-f5a300?style=flat&labelColor=0d0d10)](https://crawl4ai.com/?ref=readme-badge)

**Latest: [v0.9.4](https://github.com/unclecode/crawl4ai/releases/tag/v0.9.4) (23 Sep 2026)** · [all releases →](https://github.com/unclecode/crawl4ai/releases)
**Latest: [v0.9.5](https://github.com/unclecode/crawl4ai/releases/tag/v0.9.5) (14 Oct 2026)** · [all releases →](https://github.com/unclecode/crawl4ai/releases)

<a href="https://crawl4ai.com/?ref=readme-banner">
<picture>
Expand Down
4 changes: 4 additions & 0 deletions SECURITY-CREDITS.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,3 +25,7 @@ We thank the following security researchers for their responsible disclosure:
| arpe1618 | GitHub: [arpe1618](https://github.com/arpe1618) | Blind SSRF via the robots.txt fetch in RobotsParser.can_fetch bypassing the Docker egress controls (0.9.4) | 2026-09-04 |
| Ibrahim AlJaafreh - Cystack RedTeam | [LinkedIn](https://www.linkedin.com/in/ibrahim-aljaafreh-glitch/), [cystack.ps](https://cystack.ps) | SSRF with response disclosure via link_preview_config through the URL seeder (0.9.4) | 2026-09-04 |
| Adam Jordan | GitHub: [adamyordan](https://github.com/adamyordan) | Untrusted-config gate bypass via dict-wrapper laundering, leaking server env vars (0.9.4) | 2026-09-08 |
| arpe1618 | GitHub: [arpe1618](https://github.com/arpe1618) | Blind SSRF via the CacheValidator freshness check bypassing the Docker egress proxy (0.9.5) | 2026-09-29 |
| Zhixi "Jace" Sun | GitHub: [manus-use](https://github.com/manus-use) | Page JavaScript injection via css_selector and wait_for, HuggingFace model choice via CosineStrategy.model_name, and ReDoS via untrusted regex patterns on the Docker server (0.9.5) | 2026-10-01 |
| Chunyi Liu | GitHub: [liuchunyi-buaa](https://github.com/liuchunyi-buaa) | SQL injection in the docs marketplace backend, DOM XSS in the LinkedIn graph view example, unbounded delay fields in untrusted config bodies, and Docker hardening gaps (header-name validation, DomainFilter host parsing, silent guard failures) (0.9.5) | 2026-09-21 |
| Julian Godfrey | Email report | Chromium launched with --ignore-certificate-errors so TLS verification was off, and the Chromium sandbox opt-in had no effect (0.9.5) | 2026-10-04 |
19 changes: 19 additions & 0 deletions SECURITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -96,6 +96,17 @@ When using Crawl4AI as a Python library:

All issues below are fixed in the version named. Advisories with a GHSA id are published under [Security Advisories](https://github.com/unclecode/crawl4ai/security/advisories). Full detail for every release is in [CHANGELOG.md](CHANGELOG.md); reporter credits are in [SECURITY-CREDITS.md](SECURITY-CREDITS.md).

### Fixed in v0.9.5 (2026-10-14)

| ID | Severity | Component | Description | Fix |
|----|----------|-----------|-------------|-----|
| GHSA-w4xc-px5p-c32w | MEDIUM | Library + Docker API | Blind SSRF via the cache freshness check in `CacheValidator.validate`, outside the egress proxy (CWE-918) | Fetch routed through the pinning egress proxy |
| GHSA-qj46-wp7m-rgj8 | MEDIUM | Docker API | Page JavaScript injection via untrusted `css_selector` and `wait_for` (CWE-95) | `css_selector` passed to `evaluate()` as an argument; JavaScript `wait_for` refused for untrusted bodies |
| GHSA-f856-7xp9-w3cc | HIGH | Docker API | Untrusted `CosineStrategy.model_name` picks the HuggingFace model the server loads; code execution with PyTorch < 2.6 (CWE-502) | `model_name` refused from untrusted bodies |
| GHSA-j843-8p47-rhxp | MEDIUM | Docker API | ReDoS via caller-supplied regex patterns freezes the server (CWE-1333) | `RegexExtractionStrategy.custom`, `RegexChunking.patterns`, and schema `regex` fields refused from untrusted bodies |
| GHSA-rhjm-7hj7-75fx | MEDIUM | Docker API | Unbounded `delay_before_return_html`, `mean_delay`, `max_range` in untrusted bodies (CWE-770) | Capped at 60 seconds |
| GHSA-j2qh-f73r-mg3q | MEDIUM | Library + Docker API | Chromium launched with `--ignore-certificate-errors`, so TLS verification was off; sandbox opt-in ignored (CWE-295) | Flag removed; insecure mode only from `ignore_https_errors`; `CRAWL4AI_CHROMIUM_SANDBOX` reaches every browser |

### Fixed in v0.9.4 (2026-09-23)

| ID | Severity | Component | Description | Fix |
Expand Down Expand Up @@ -199,6 +210,14 @@ Secure-by-default rework of the Docker API server. The pip library is unchanged.

## Security Features

### v0.9.5+

- **Cache freshness check through the egress proxy**: `CacheValidator` uses the same pinning proxy as the seeder and robots.txt
- **More untrusted fields refused**: JavaScript `wait_for`, `CosineStrategy.model_name`, and caller-supplied regexes
- **Delay caps**: `delay_before_return_html`, `mean_delay`, and `max_range` capped at 60 seconds for untrusted bodies
- **TLS verification on in Chromium**: `--ignore-certificate-errors` no longer in the default launch flags
- **Fail-closed egress pinning**: a crawl that cannot be pinned to the egress proxy is refused

### v0.9.4+

- **Library egress proxy**: `crawl4ai/egress_policy.py` routes the library's own HTTP clients (URL seeder, robots.txt) through the Docker server's pinning proxy, so they get the same resolve-and-pin rule as the browser
Expand Down
2 changes: 1 addition & 1 deletion crawl4ai/__version__.py
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# crawl4ai/__version__.py

# This is the version that will be used for stable releases
__version__ = "0.9.4"
__version__ = "0.9.5"

# For nightly builds, this gets set during build process
__nightly_version__ = None
Expand Down
Loading
Loading