Skip to content

Commit 96ecd7d

Browse files
Mantisusvdusek
andcommitted
feat(http-clients)!: Add a cookies argument to send_request (#2352)
### Description `HttpClient.send_request` and `context.send_request` take a new `cookies` argument, the cookies to send and store instead of the session cookies for a single call. To send no session cookies, pass `cookies=SessionCookies()`. In `PlaywrightCrawler`, they're sent instead of the browser context cookies, and the response cookies still reach the session through the browser context. ### Issues - Part of: #2307 - Follow-up: #2385 *✍️ Drafted by Claude Code* --------- Co-authored-by: Vlada Dusek <v.dusek96@gmail.com>
1 parent b5f2f2e commit 96ecd7d

10 files changed

Lines changed: 179 additions & 37 deletions

File tree

‎docs/06_upgrading/upgrading_to_v2.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -387,13 +387,15 @@ The `headers` argument of a client still applies to every request. Its `Authoriz
387387

388388
A `Cookie` header passed with a request or set on the client is merged with the session cookies while the redirect chain stays on its origin. A cookie in the header wins over a session cookie of the same name and isn't stored in the session. In v1, `ImpitHttpClient`, `HttpxHttpClient` and `context.send_request` of `PlaywrightCrawler` sent the header instead of the session cookies, so an empty `Cookie` header kept them out. `CurlImpersonateHttpClient` sent the session cookies in a second `Cookie` header.
389389

390+
`send_request` and `context.send_request` take a new `cookies` argument, the cookies to send and store instead of the session cookies for a single call. To send no session cookies, pass `cookies=SessionCookies()`. With `PlaywrightCrawler`, the response cookies still reach the session through the browser context.
391+
390392
Without a session, or with `persist_cookies_per_session=False`, the cookies a server sets during a redirect chain are sent with the following requests of that chain. In v1, `ImpitHttpClient` and `HttpxHttpClient` didn't send them.
391393

392394
### Some client arguments are ignored
393395

394396
The following keyword arguments of the underlying libraries are now ignored with a warning:
395397

396-
- `cookies` of every client and `cookie_jar` of `ImpitHttpClient`, since cookies are kept in sessions.
398+
- `cookies` of every client and `cookie_jar` of `ImpitHttpClient`, since cookies are kept in sessions. To send other cookies with a single call, pass `cookies` to `send_request`.
397399
- `auth` and `params` of `HttpxHttpClient` and `CurlImpersonateHttpClient`, which would reach every request of a redirect chain. Pass the credentials in the `Authorization` header and the query in the URL instead.
398400
- `max_redirects` of every client and `allow_redirects` of `CurlImpersonateHttpClient`, since `send_request` follows the redirects.
399401

‎src/crawlee/_types.py‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,7 @@
2222
from crawlee.configuration import Configuration
2323
from crawlee.http_clients import HttpResponse
2424
from crawlee.proxy_configuration import ProxyInfo
25-
from crawlee.sessions import Session
25+
from crawlee.sessions import Session, SessionCookies
2626
from crawlee.storage_backends import StorageBackend
2727
from crawlee.storages import KeyValueStore
2828

@@ -581,6 +581,7 @@ def __call__(
581581
method: HttpMethod = 'GET',
582582
payload: HttpPayload | None = None,
583583
headers: HttpHeaders | dict[str, str] | None = None,
584+
cookies: SessionCookies | None = None,
584585
timeout: timedelta | None = None,
585586
) -> Coroutine[None, None, HttpResponse]:
586587
"""Call send request function.
@@ -590,6 +591,9 @@ def __call__(
590591
method: The HTTP method to use.
591592
headers: The headers to include in the request.
592593
payload: The payload to include in the request.
594+
cookies: The cookies to send and store instead of the cookies of the session. With `PlaywrightCrawler`,
595+
the response cookies are stored in the browser context as well, and so in the session after the
596+
request handler.
593597
timeout: Maximum time allowed to receive the response, including all redirects, and to read its body.
594598
Without it, the default timeout of the HTTP client applies.
595599

‎src/crawlee/crawlers/_basic/_basic_crawler.py‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -101,7 +101,7 @@
101101
from crawlee.http_clients import HttpClient, HttpResponse
102102
from crawlee.proxy_configuration import ProxyConfiguration, ProxyInfo
103103
from crawlee.request_loaders import RequestManager
104-
from crawlee.sessions import Session
104+
from crawlee.sessions import Session, SessionCookies
105105
from crawlee.statistics import FinalStatistics
106106
from crawlee.storage_backends import StorageBackend
107107
from crawlee.storage_backends.models import DatasetItemsListPage
@@ -1390,6 +1390,7 @@ async def send_request(
13901390
method: HttpMethod = 'GET',
13911391
payload: HttpPayload | None = None,
13921392
headers: HttpHeaders | dict[str, str] | None = None,
1393+
cookies: SessionCookies | None = None,
13931394
timeout: timedelta | None = None,
13941395
) -> HttpResponse:
13951396
response = await self._http_client.send_request(
@@ -1398,6 +1399,7 @@ async def send_request(
13981399
payload=payload,
13991400
headers=headers,
14001401
session=session,
1402+
cookies=cookies,
14011403
proxy_info=proxy_info,
14021404
timeout=timeout,
14031405
)

‎src/crawlee/crawlers/_playwright/_playwright_http_client.py‎

Lines changed: 36 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -8,8 +8,7 @@
88

99
from crawlee.crawlers._playwright._types import PlaywrightHttpResponse
1010
from crawlee.http_clients import HttpClient
11-
from crawlee.sessions import Session
12-
from crawlee.sessions._cookies import PlaywrightCookieParam
11+
from crawlee.sessions._cookies import PlaywrightCookieParam, SessionCookies
1312

1413
if TYPE_CHECKING:
1514
from collections.abc import AsyncGenerator
@@ -20,10 +19,14 @@
2019
from crawlee._types import HttpHeaders, HttpMethod, HttpPayload
2120
from crawlee.http_clients import HttpResponse
2221
from crawlee.proxy_configuration import ProxyInfo
22+
from crawlee.sessions import Session
2323

2424

2525
_browser_page_context_var: contextvars.ContextVar[Page | None] = contextvars.ContextVar('browser_context', default=None)
2626

27+
# Set while `send_request` sends the cookies passed by the caller instead of the cookies of the browser context.
28+
_caller_cookies_var: contextvars.ContextVar[bool] = contextvars.ContextVar('caller_cookies', default=False)
29+
2730

2831
@asynccontextmanager
2932
async def browser_page_context(page: Page) -> AsyncGenerator[None, None]:
@@ -55,33 +58,39 @@ async def send_request(
5558
headers: HttpHeaders | dict[str, str] | None = None,
5659
payload: HttpPayload | None = None,
5760
session: Session | None = None,
61+
cookies: SessionCookies | None = None,
5862
proxy_info: ProxyInfo | None = None,
5963
timeout: timedelta | None = None,
6064
) -> HttpResponse:
6165
# TODO: Use `session` to restore all the fingerprint headers according to the `BrowserContext`, after resolved
6266
# https://github.com/apify/crawlee-python/issues/1055
6367

6468
page = _browser_page_context_var.get()
65-
66-
# The browser context holds the cookies of the session, added before the navigation, with the changes the page
67-
# made since. The browser stores the response cookies there too, and the crawler saves them to the session after
68-
# the request handler.
69-
if page is not None:
70-
context_cookies = await page.context.cookies()
71-
session = Session()
72-
session.cookies.set_cookies_from_playwright_format(
73-
[PlaywrightCookieParam(**cookie) for cookie in context_cookies]
69+
token = _caller_cookies_var.set(cookies is not None)
70+
71+
try:
72+
# The browser context holds the cookies of the session, added before the navigation, with the changes the
73+
# page made since. The browser stores the response cookies there too, and the crawler saves them to the
74+
# session after the request handler.
75+
if page is not None and cookies is None:
76+
context_cookies = await page.context.cookies()
77+
cookies = SessionCookies()
78+
cookies.set_cookies_from_playwright_format(
79+
[PlaywrightCookieParam(**cookie) for cookie in context_cookies]
80+
)
81+
82+
return await super().send_request(
83+
url,
84+
method=method,
85+
headers=headers,
86+
payload=payload,
87+
session=session,
88+
cookies=cookies,
89+
proxy_info=proxy_info,
90+
timeout=timeout,
7491
)
75-
76-
return await super().send_request(
77-
url,
78-
method=method,
79-
headers=headers,
80-
payload=payload,
81-
session=session,
82-
proxy_info=proxy_info,
83-
timeout=timeout,
84-
)
92+
finally:
93+
_caller_cookies_var.reset(token)
8594

8695
@override
8796
async def _fetch(
@@ -100,11 +109,16 @@ async def _fetch(
100109
if browser_context is None:
101110
raise RuntimeError('Unable to create an `APIRequestContext` outside the browser context')
102111

112+
fetch_headers = dict(headers)
113+
if _caller_cookies_var.get():
114+
# Without a `Cookie` header, Playwright sends the cookies of the browser context.
115+
fetch_headers.setdefault('cookie', '')
116+
103117
# Proxies appropriate to the browser context are used
104118
response = await browser_context.request.fetch(
105119
url_or_request=url,
106120
method=method.lower(),
107-
headers=dict(headers) or None,
121+
headers=fetch_headers or None,
108122
data=payload,
109123
timeout=timeout.total_seconds() * 1000 if timeout is not None else None,
110124
# Redirects are followed hop by hop by `send_request`.

‎src/crawlee/http_clients/_base.py‎

Lines changed: 11 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -226,6 +226,7 @@ async def send_request(
226226
headers: HttpHeaders | dict[str, str] | None = None,
227227
payload: HttpPayload | None = None,
228228
session: Session | None = None,
229+
cookies: SessionCookies | None = None,
229230
proxy_info: ProxyInfo | None = None,
230231
timeout: timedelta | None = None,
231232
) -> HttpResponse:
@@ -246,6 +247,8 @@ async def send_request(
246247
session: The session whose cookies are sent. The response cookies are stored in it when
247248
`persist_cookies_per_session` is enabled. Otherwise, and without a session, they are kept only for the
248249
redirect chain of this request.
250+
cookies: The cookies to send and store instead of the cookies of `session`, even with
251+
`persist_cookies_per_session` disabled.
249252
proxy_info: The information about the proxy to be used.
250253
timeout: Maximum time allowed to receive the response, including all redirects, and to read its body.
251254
Without it, the default timeout of the HTTP client applies.
@@ -263,13 +266,14 @@ async def send_request(
263266
# The headers of the client go through the redirect rules below too, so its credentials never leave the origin.
264267
headers = self._headers | HttpHeaders(headers or {})
265268

266-
if session is None:
267-
cookies = SessionCookies()
268-
elif self._persist_cookies_per_session:
269-
cookies = session.cookies
270-
else:
271-
# The cookies of the session are still sent, but the response cookies go to a copy.
272-
cookies = SessionCookies(session.cookies)
269+
if cookies is None:
270+
if session is None:
271+
cookies = SessionCookies()
272+
elif self._persist_cookies_per_session:
273+
cookies = session.cookies
274+
else:
275+
# The cookies of the session are still sent, but the response cookies go to a copy.
276+
cookies = SessionCookies(session.cookies)
273277

274278
# `encoded=True` keeps the path and the query byte for byte, including a safe `%2F` in query.
275279
current_url = _with_ascii_host(URL(url, encoded=True))

‎src/crawlee/http_clients/_httpx.py‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,7 @@
2626

2727
from crawlee._types import HttpMethod, HttpPayload
2828
from crawlee.proxy_configuration import ProxyInfo
29-
from crawlee.sessions import Session
29+
from crawlee.sessions import Session, SessionCookies
3030

3131
logger = getLogger(__name__)
3232

@@ -182,6 +182,7 @@ async def send_request(
182182
headers: HttpHeaders | dict[str, str] | None = None,
183183
payload: HttpPayload | None = None,
184184
session: Session | None = None,
185+
cookies: SessionCookies | None = None,
185186
proxy_info: ProxyInfo | None = None,
186187
timeout: timedelta | None = None,
187188
) -> HttpResponse:
@@ -196,6 +197,7 @@ async def send_request(
196197
headers=self._combine_headers(headers),
197198
payload=payload,
198199
session=session,
200+
cookies=cookies,
199201
proxy_info=proxy_info,
200202
timeout=timeout,
201203
)

‎tests/integration/crawlers/_playwright/test_playwright_crawler.py‎

Lines changed: 19 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -35,7 +35,7 @@
3535
from crawlee.fingerprint_suite._header_generator import fingerprint_browser_type_from_playwright_browser_type
3636
from crawlee.http_clients import ImpitHttpClient
3737
from crawlee.proxy_configuration import ProxyConfiguration
38-
from crawlee.sessions import Session, SessionPool
38+
from crawlee.sessions import CookieParam, Session, SessionCookies, SessionPool
3939
from crawlee.statistics import Statistics
4040
from crawlee.statistics._error_snapshotter import ErrorSnapshotter
4141
from crawlee.storages import RequestQueue
@@ -604,6 +604,24 @@ async def second_handler(context: PlaywrightCrawlingContext) -> None:
604604
assert sent_cookies == [{}]
605605

606606

607+
async def test_send_request_cookies_argument_replaces_page_cookies(server_url: URL) -> None:
608+
"""Test that `context.send_request` with `cookies` sends none of the cookies of the browser context."""
609+
crawler = PlaywrightCrawler()
610+
sent_cookies: list[dict[str, str]] = []
611+
612+
@crawler.router.default_handler
613+
async def request_handler(context: PlaywrightCrawlingContext) -> None:
614+
cookies = SessionCookies([CookieParam(name='from_argument', value='1', domain=server_url.host or '')])
615+
matching = await context.send_request(str(server_url / 'cookies'), cookies=cookies)
616+
sent_cookies.append(json.loads(await matching.read())['cookies'])
617+
empty = await context.send_request(str(server_url / 'cookies'), cookies=SessionCookies())
618+
sent_cookies.append(json.loads(await empty.read())['cookies'])
619+
620+
await crawler.run([str(server_url.with_path('set_cookies').extend_query(a=1))])
621+
622+
assert sent_cookies == [{'from_argument': '1'}, {}]
623+
624+
607625
async def test_read_write_cookies(server_url: URL) -> None:
608626
"""Test that cookies are reloaded correctly."""
609627
async with SessionPool(max_pool_size=1) as session_pool:

‎tests/integration/http_clients/test_http_clients.py‎

Lines changed: 38 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@
1313

1414
from crawlee.errors import ProxyError
1515
from crawlee.http_clients import CurlImpersonateHttpClient, HttpClient, HttpxHttpClient, ImpitHttpClient
16-
from crawlee.sessions import CookieParam, Session
16+
from crawlee.sessions import CookieParam, Session, SessionCookies
1717
from tests.integration.server import generate_file_content
1818
from tests.integration.server_endpoints import HELLO_WORLD
1919

@@ -499,6 +499,43 @@ async def test_cookies_not_persisted(custom_http_client: HttpClient, server_url:
499499
assert session.cookies.get_cookies_as_dicts() == []
500500

501501

502+
async def test_cookies_argument_replaces_session_cookies(http_client: HttpClient, server_url: URL) -> None:
503+
"""Test that the cookies passed in `cookies` are sent and stored instead of the cookies of the session."""
504+
session = Session(cookies=[CookieParam(name='from_session', value='1', domain=server_url.host or '')])
505+
cookies = SessionCookies([CookieParam(name='from_argument', value='1', domain=server_url.host or '')])
506+
507+
async with await http_client.send_request(
508+
str((server_url / 'set_cookies').with_query(a='1')),
509+
session=session,
510+
cookies=cookies,
511+
) as response:
512+
assert (await read_json(response))['cookies'] == {'from_argument': '1', 'a': '1'}
513+
514+
assert {item['name'] for item in cookies.get_cookies_as_dicts()} == {'from_argument', 'a'}
515+
assert {item['name'] for item in session.cookies.get_cookies_as_dicts()} == {'from_session'}
516+
517+
518+
@pytest.mark.parametrize(
519+
'custom_http_client',
520+
[
521+
pytest.param(lambda: CurlImpersonateHttpClient(persist_cookies_per_session=False), id='curl'),
522+
pytest.param(lambda: HttpxHttpClient(persist_cookies_per_session=False), id='httpx'),
523+
pytest.param(lambda: ImpitHttpClient(persist_cookies_per_session=False), id='impit'),
524+
],
525+
indirect=['custom_http_client'],
526+
)
527+
async def test_cookies_argument_stored_when_not_persisted(custom_http_client: HttpClient, server_url: URL) -> None:
528+
"""Test that the response cookies are stored in `cookies` even with `persist_cookies_per_session` disabled."""
529+
cookies = SessionCookies()
530+
531+
async with await custom_http_client.send_request(
532+
str((server_url / 'set_cookies').with_query(a='1')), cookies=cookies
533+
):
534+
pass
535+
536+
assert {item['name'] for item in cookies.get_cookies_as_dicts()} == {'a'}
537+
538+
502539
@pytest.mark.parametrize(
503540
'custom_http_client',
504541
[

‎tests/unit/crawlers/_basic/test_basic_crawler.py‎

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,7 @@
2525
from crawlee._log_config import CrawleeLogFormatter
2626
from crawlee._request import Request, RequestState
2727
from crawlee._types import BasicCrawlingContext, EnqueueLinksKwargs
28+
from crawlee._utils.robots import RobotsTxtFile
2829
from crawlee.autoscaling import ConcurrencyConsumer, ConcurrencySystem
2930
from crawlee.configuration import Configuration
3031
from crawlee.crawlers import BasicCrawler
@@ -48,7 +49,7 @@
4849
StalledStatus,
4950
ThrottlingRequestManager,
5051
)
51-
from crawlee.sessions import Session, SessionPool
52+
from crawlee.sessions import Session, SessionCookies, SessionPool
5253
from crawlee.statistics import FinalStatistics, Statistics, StatisticsState
5354
from crawlee.storage_backends import FileSystemStorageBackend, MemoryStorageBackend
5455
from crawlee.storages import Dataset, KeyValueStore, RequestQueue
@@ -611,12 +612,14 @@ async def failed_request_handler(context: BasicCrawlingContext, error: Exception
611612
await crawler.run(['https://a.placeholder.com', 'https://b.placeholder.com', 'https://c.placeholder.com'])
612613

613614

614-
async def test_send_request_forwards_timeout_to_http_client() -> None:
615+
async def test_send_request_forwards_arguments_to_http_client() -> None:
616+
"""Test that `context.send_request` passes all its arguments to the HTTP client."""
615617
http_client = AsyncMock(spec=HttpClient)
616618
response = Mock(spec=HttpResponse)
617619
http_client.send_request.return_value = response
618620
crawler = BasicCrawler(http_client=http_client, use_session_pool=False, max_requests_per_crawl=1)
619621
timeout = timedelta(seconds=12)
622+
cookies = SessionCookies({'name': 'value'})
620623
result = None
621624

622625
@crawler.router.default_handler
@@ -627,6 +630,7 @@ async def handler(context: BasicCrawlingContext) -> None:
627630
method='POST',
628631
payload=b'Hello, world!',
629632
headers={'Content-Type': 'text/plain'},
633+
cookies=cookies,
630634
timeout=timeout,
631635
)
632636

@@ -639,6 +643,7 @@ async def handler(context: BasicCrawlingContext) -> None:
639643
payload=b'Hello, world!',
640644
headers={'Content-Type': 'text/plain'},
641645
session=None,
646+
cookies=cookies,
642647
proxy_info=None,
643648
timeout=timeout,
644649
)

0 commit comments

Comments
 (0)