Skip to content

Commit 3132135

Browse files
committed
refactor!: make RequestQueueClient.is_finished abstract (#2317)
Makes `RequestQueueClient.is_finished` abstract. The default fell back to `is_empty`, which ignores requests still in progress, so a custom client could report its queue as finished too early. All built-in clients and the Apify SDK clients already implement the method. The v2 upgrade guide tells custom client authors to implement it. Closes: #1985 *✍️ Drafted by Claude Code*
1 parent 0905964 commit 3132135

2 files changed

Lines changed: 5 additions & 9 deletions

File tree

‎docs/06_upgrading/upgrading_to_v2.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,3 +16,7 @@ We drop support for Python 3.10, which reached its end of life in October 2026.
1616
## `StatisticsState.crawler_runtime` is read-only
1717

1818
`StatisticsState.crawler_runtime` is computed from the crawler's start and finish timestamps. In v1, assigning to it only emitted a `DeprecationWarning` and the value was ignored. In v2, the property has no setter, so an assignment raises `AttributeError`. Remove any such assignments.
19+
20+
## `RequestQueueClient.is_finished` is abstract
21+
22+
`RequestQueueClient.is_finished` is now an abstract method. In v1, it fell back to `is_empty`, which ignores requests still being processed and could report the queue as finished too early. A subclass that doesn't implement it raises `TypeError` on instantiation. If you have a custom request queue client, implement `is_finished` so that it returns `True` only when the queue is empty and no requests are in progress. The built-in clients already implement it.

‎src/crawlee/storage_clients/_base/_request_queue_client.py‎

Lines changed: 1 addition & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -125,20 +125,12 @@ async def is_empty(self) -> bool:
125125
True if the request queue is empty, False otherwise.
126126
"""
127127

128+
@abstractmethod
128129
async def is_finished(self) -> bool:
129130
"""Check if the request queue is finished.
130131
131132
A finished queue is empty and has no requests currently being processed.
132133
133-
Warning:
134-
This default only checks `is_empty`, which reports whether requests are available to fetch, not
135-
whether requests are still being processed. It can therefore return `True` while requests are in
136-
progress. Subclasses that track in-progress requests should override this method; all built-in
137-
clients already do.
138-
139134
Returns:
140135
True if the request queue is finished, False otherwise.
141136
"""
142-
# TODO: Make this method abstract.
143-
# https://github.com/apify/crawlee-python/issues/1985
144-
return await self.is_empty()

0 commit comments

Comments
 (0)