Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions docs/upgrading/upgrading_to_v2.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,3 +16,7 @@ We drop support for Python 3.10, which reached its end of life in October 2026.
## `StatisticsState.crawler_runtime` is read-only

`StatisticsState.crawler_runtime` is computed from the crawler's start and finish timestamps. In v1, assigning to it only emitted a `DeprecationWarning` and the value was ignored. In v2, the property has no setter, so an assignment raises `AttributeError`. Remove any such assignments.

## `RequestQueueClient.is_finished` is abstract

`RequestQueueClient.is_finished` is now an abstract method. In v1, it fell back to `is_empty`, which ignores requests still being processed and could report the queue as finished too early. A subclass that doesn't implement it raises `TypeError` on instantiation. If you have a custom request queue client, implement `is_finished` so that it returns `True` only when the queue is empty and no requests are in progress. The built-in clients already implement it.
10 changes: 1 addition & 9 deletions src/crawlee/storage_clients/_base/_request_queue_client.py
Original file line number Diff line number Diff line change
Expand Up @@ -125,20 +125,12 @@ async def is_empty(self) -> bool:
True if the request queue is empty, False otherwise.
"""

@abstractmethod
async def is_finished(self) -> bool:
"""Check if the request queue is finished.

A finished queue is empty and has no requests currently being processed.

Warning:
This default only checks `is_empty`, which reports whether requests are available to fetch, not
whether requests are still being processed. It can therefore return `True` while requests are in
progress. Subclasses that track in-progress requests should override this method; all built-in
clients already do.

Returns:
True if the request queue is finished, False otherwise.
"""
# TODO: Make this method abstract.
# https://github.com/apify/crawlee-python/issues/1985
return await self.is_empty()
Loading