Skip to content

Commit b5f2f2e

Browse files
Mantisusvdusek
andcommitted
feat(http-clients)!: Merge a Cookie header with stored cookies (#2385)
### Description A `Cookie` header passed with a request or set on the client is now merged with the session cookies, with every HTTP client and in `context.send_request` of `PlaywrightCrawler`. A cookie in the header wins over a session cookie of the same name and isn't stored in the session. An empty `Cookie` header no longer keeps the session cookies out. The v2 upgrade guide describes the change. ### Issues - Part of: #2307 - Follow-up: #2349 *✍️ Drafted by Claude Code* Co-authored-by: Vlada Dusek <v.dusek96@gmail.com>
1 parent d7ad65a commit b5f2f2e

5 files changed

Lines changed: 70 additions & 19 deletions

File tree

‎docs/06_upgrading/upgrading_to_v2.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -383,6 +383,12 @@ The crawlers pass the remaining `navigation_timeout` to the HTTP client, so it b
383383

384384
The `headers` argument of a client still applies to every request. Its `Authorization`, `Proxy-Authorization` and `Cookie` headers are dropped when a redirect leaves the origin, the same as the headers passed with a request. With `HttpxHttpClient`, the headers of the client now win over the generated browser headers. In v1, the generated browser headers replaced them.
385385

386+
### A `Cookie` header is merged with the session cookies
387+
388+
A `Cookie` header passed with a request or set on the client is merged with the session cookies while the redirect chain stays on its origin. A cookie in the header wins over a session cookie of the same name and isn't stored in the session. In v1, `ImpitHttpClient`, `HttpxHttpClient` and `context.send_request` of `PlaywrightCrawler` sent the header instead of the session cookies, so an empty `Cookie` header kept them out. `CurlImpersonateHttpClient` sent the session cookies in a second `Cookie` header.
389+
390+
Without a session, or with `persist_cookies_per_session=False`, the cookies a server sets during a redirect chain are sent with the following requests of that chain. In v1, `ImpitHttpClient` and `HttpxHttpClient` didn't send them.
391+
386392
### Some client arguments are ignored
387393

388394
The following keyword arguments of the underlying libraries are now ignored with a warning:

‎src/crawlee/http_clients/_base.py‎

Lines changed: 20 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -82,11 +82,27 @@ def _redirect_method(status_code: int, method: HttpMethod) -> HttpMethod:
8282
return method
8383

8484

85+
def _split_cookie_header(value: str) -> list[tuple[str, str]]:
86+
"""Split a `Cookie` header into `(name, pair)` tuples, keeping each pair as it was written."""
87+
pairs = [pair.strip() for pair in value.split(';')]
88+
return [(pair.partition('=')[0].strip(), pair) for pair in pairs if pair]
89+
90+
8591
def _add_cookie_header(headers: HttpHeaders, cookies: SessionCookies, url: str) -> HttpHeaders:
86-
"""Add the stored cookies matching `url` as the `Cookie` header, unless the headers already have one."""
87-
if 'cookie' in headers or not (cookie_header := cookies.get_cookie_string(url)):
92+
"""Add the stored cookies matching `url` to the `Cookie` header.
93+
94+
A cookie already in the header wins over a stored cookie of the same name.
95+
"""
96+
explicit_cookies = _split_cookie_header(headers.get('cookie', ''))
97+
explicit_names = {name for name, _ in explicit_cookies}
98+
stored_cookie_string = cookies.get_cookie_string(url)
99+
stored_pairs = [pair for name, pair in _split_cookie_header(stored_cookie_string) if name not in explicit_names]
100+
101+
if not stored_pairs:
88102
return headers
89103

104+
explicit_pairs = [pair for _, pair in explicit_cookies]
105+
cookie_header = '; '.join(stored_pairs + explicit_pairs)
90106
return headers | HttpHeaders({'cookie': cookie_header})
91107

92108

@@ -216,8 +232,8 @@ async def send_request(
216232
"""Send an HTTP request via the client, following up to 10 redirects unless `follow_redirects` is disabled.
217233
218234
Every request of the redirect chain sends the cookies matching its URL and stores the cookies of its response. A
219-
`Cookie` header passed by the caller is sent instead of the stored cookies, and its cookies are not stored. The
220-
`Authorization`, `Proxy-Authorization` and `Cookie` headers are dropped when a redirect leaves the origin.
235+
cookie in the `Cookie` header passed by the caller wins over a stored cookie of the same name and is not stored.
236+
The `Authorization`, `Proxy-Authorization` and `Cookie` headers are dropped when a redirect leaves the origin.
221237
222238
The body of the returned response is not read yet. Read it, or call `aclose` on the response to release its
223239
connection.

‎tests/integration/crawlers/_playwright/test_playwright_crawler.py‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -562,6 +562,28 @@ async def request_handler(context: PlaywrightCrawlingContext) -> None:
562562
assert sent_cookies == {'a': '1', 'b': '2', 'check': 'test'}
563563

564564

565+
async def test_send_request_merges_cookie_header_with_page_cookies(server_url: URL) -> None:
566+
"""Test that `context.send_request` merges the `Cookie` header with the page cookies and the header wins."""
567+
crawler = PlaywrightCrawler()
568+
sent_cookies: list[list[str]] = []
569+
570+
@crawler.router.default_handler
571+
async def request_handler(context: PlaywrightCrawlingContext) -> None:
572+
await context.page.context.add_cookies(
573+
[
574+
{'name': 'manual', 'value': 'from_page', 'url': str(server_url)},
575+
{'name': 'from_page', 'value': '1', 'url': str(server_url)},
576+
]
577+
)
578+
response = await context.send_request(str(server_url / 'headers'), headers={'cookie': 'manual=value'})
579+
cookie_header = json.loads(await response.read())['cookie']
580+
sent_cookies.append(sorted(cookie_header.split('; ')))
581+
582+
await crawler.run([str(server_url)])
583+
584+
assert sent_cookies == [['from_page=1', 'manual=value']]
585+
586+
565587
async def test_send_request_skips_cookies_deleted_by_page(server_url: URL) -> None:
566588
"""Test that `context.send_request` doesn't send a session cookie that the page deleted in the handler."""
567589
crawler = PlaywrightCrawler(session_pool=SessionPool(max_pool_size=1))

‎tests/integration/http_clients/test_http_clients.py‎

Lines changed: 19 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -544,15 +544,22 @@ async def test_cookie_header_rebuilt_per_hop(http_client: HttpClient, server_url
544544

545545

546546
async def test_cookie_header_wins_over_session(http_client: HttpClient, server_url: URL) -> None:
547-
"""Test that a `Cookie` header passed by the caller replaces the cookies of the session."""
548-
session = Session(cookies=[CookieParam(name='from_jar', value='1', domain=server_url.host or '')])
547+
"""Test that the `Cookie` header of the caller is merged with the session cookies and wins on a shared name."""
548+
session = Session(
549+
cookies=[
550+
CookieParam(name='manual', value='from_jar', domain=server_url.host or ''),
551+
CookieParam(name='from_jar', value='1', domain=server_url.host or ''),
552+
]
553+
)
549554

550555
async with await http_client.send_request(
551-
str(server_url / 'cookies'),
556+
str(server_url / 'headers'),
552557
session=session,
553558
headers={'cookie': 'manual=value'},
554559
) as response:
555-
assert (await read_json(response))['cookies'] == {'manual': 'value'}
560+
cookie_header = (await read_json(response))['cookie']
561+
562+
assert sorted(cookie_header.split('; ')) == ['from_jar=1', 'manual=value']
556563

557564

558565
async def test_cookie_header_not_stored_in_session(http_client: HttpClient, server_url: URL) -> None:
@@ -605,30 +612,30 @@ async def test_session_cookies_take_over_cross_origin(
605612

606613

607614
async def test_cookie_header_wins_over_session_on_redirect(http_client: HttpClient, server_url: URL) -> None:
608-
"""Test that a `Cookie` header of the caller keeps beating the session cookies after a redirect."""
609-
session = Session(cookies=[CookieParam(name='from_jar', value='1', domain=server_url.host or '')])
610-
redirect_url = (server_url / 'redirect').with_query(url=str(server_url / 'cookies'), status=302)
615+
"""Test that a cookie in the `Cookie` header of the caller keeps beating the session cookie after a redirect."""
616+
session = Session(cookies=[CookieParam(name='manual', value='from_jar', domain=server_url.host or '')])
617+
redirect_url = (server_url / 'redirect').with_query(url=str(server_url / 'headers'), status=302)
611618

612619
async with await http_client.send_request(
613620
str(redirect_url),
614621
session=session,
615622
headers={'cookie': 'manual=value'},
616623
) as response:
617-
assert (await read_json(response))['cookies'] == {'manual': 'value'}
624+
assert (await read_json(response))['cookie'] == 'manual=value'
618625

619626

620-
async def test_empty_cookie_header_suppresses_session_cookies(http_client: HttpClient, server_url: URL) -> None:
621-
"""Test that an empty `Cookie` header of the caller keeps the session cookies out of every hop."""
627+
async def test_empty_cookie_header_keeps_session_cookies(http_client: HttpClient, server_url: URL) -> None:
628+
"""Test that the session cookies reach every hop despite an empty `Cookie` header of the caller."""
622629
session = Session(cookies=[CookieParam(name='from_jar', value='1', domain=server_url.host or '')])
623630
redirect_url = (server_url / 'redirect').with_query(url=str(server_url / 'cookies'), status=302)
624631

625632
async with await http_client.send_request(
626633
str(server_url / 'cookies'), session=session, headers={'cookie': ''}
627634
) as direct:
628-
assert (await read_json(direct))['cookies'] == {}
635+
assert (await read_json(direct))['cookies'] == {'from_jar': '1'}
629636

630637
async with await http_client.send_request(str(redirect_url), session=session, headers={'cookie': ''}) as redirected:
631-
assert (await read_json(redirected))['cookies'] == {}
638+
assert (await read_json(redirected))['cookies'] == {'from_jar': '1'}
632639

633640

634641
async def test_auth_kept_same_origin(http_client: HttpClient, server_url: URL) -> None:

‎tests/integration/http_clients/test_httpx.py‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -123,8 +123,8 @@ async def test_client_headers_win_over_generated(server_url: URL) -> None:
123123
assert headers['user-agent'] == 'client'
124124

125125

126-
async def test_client_cookie_header_wins_over_session(server_url: URL) -> None:
127-
"""Test that a `Cookie` header set on the underlying client replaces the cookies of the session."""
126+
async def test_client_cookie_header_merged_with_session(server_url: URL) -> None:
127+
"""Test that a `Cookie` header set on the client is sent along with the cookies of the session."""
128128
session = Session(cookies=[CookieParam(name='from_jar', value='1', domain=server_url.host or '')])
129129

130130
async with (
@@ -133,7 +133,7 @@ async def test_client_cookie_header_wins_over_session(server_url: URL) -> None:
133133
):
134134
cookies = (await read_json(response))['cookies']
135135

136-
assert cookies == {'from_client': '1'}
136+
assert cookies == {'from_jar': '1', 'from_client': '1'}
137137

138138

139139
async def test_cookies_kwarg_ignored(server_url: URL) -> None:

0 commit comments

Comments
 (0)