refactor(request-loaders)!: Replace is_empty and is_finished with check_readiness

[Mantisus]

Description

RequestLoader and RequestManager replace is_empty() and is_finished() with a single check_readiness(). It returns one of four statuses, exported from crawlee.request_loaders: ReadyStatus, WaitingStatus, StalledStatus or FinishedStatus. WaitingStatus.ready_at says when a source expects to have work again, and ThrottlingRequestManager sets it from its cooldowns. StalledStatus.reason says why a source can't make progress. Storage clients keep is_empty() and is_finished(). The upgrade guide covers the migration.

• ThrottlingRequestManager.get_stall_reason() is removed. The crawler reads a stall from the status, so it now also detects a stalled domain when the ThrottlingRequestManager is wrapped in a RequestManagerTandem.
• RequestManagerTandem moves a request from its loader only when the loader is ready, so a SitemapRequestLoader that's still loading sitemaps no longer holds up the requests already in the manager.
• RequestLoader.fetch_next_request() no longer has to wait for a request. A None means nothing is pending right now, not that the loader is finished. SitemapRequestLoader.fetch_next_request() returns None once its sitemaps are loaded, instead of waiting until every fetched request is handled.

Issues

• Part of: Replace is_empty/is_finished with check_readiness() and add pacing signals to RequestManager #2296

✍️ Drafted by Claude Code

[vdusek]

Oh, this is quite less convenient than how it worked before 😕.

[vdusek]

ok, then let's use literal comparison at least, so users aren't pushed into an extra import

[vdusek]

As I mentioned above, this is much less convenient than before. You now need an additional import and an isinstance() check.

Maybe we could introduce helper methods for checking statuses, so determining whether an RQ is in a particular status would be a single method call?

[vdusek]

ok, then let's use literal comparison at least, so users aren't pushed into an extra import

[vdusek]

@janbuchar Have you considered keeping is_finished as a simple helper that wraps

isinstance(await rq.check_readiness(), FinishedStatus)

?

We could add similar helpers for the other statuses where they make sense.

I mean, the new API is a bit worse in terms of ergonomics (maybe with AI agents it doesn't matter that much?).

[Mantisus]

ok, then let's use literal comparison at least, so users aren't pushed into an extra import

Implemented in 883a68c