Skip to content

Commit 6aa8ead

Browse files
authored
feat: Add live iteration over a run's dataset items (#1079)
Adds `RunClient.iterate_dataset_items()` and its async twin. They yield a run's dataset items while the run is still pushing them and return once the run has finished and the dataset is drained. Each poll reads the dataset's `itemCount` and fetches pages whose `limit` ends at it. The endpoint scans exactly `limit` rows, so offsets stay exact even when `clean`, `skip_empty` or `unwind` change how many items a page returns. Between polls, `wait_for_finish()` waits up to `poll_interval` for the run to finish. `itemCount` lags about 5 s, so after the run finishes, the rows past it are read page by page until a page comes back empty. With `clean`, `skip_empty` or `unwind`, one unfiltered `limit=1` read confirms the end. Polling goes on through `ABORTING` and `TIMING-OUT`, since such a run can still push items. The arguments follow `DatasetClient.iterate_items` without `desc` and `signature`, plus `poll_interval`. Docs: a new section in the Retrieve Actor data guide, and a mention on the convenience methods and pagination pages. Closes: #1065 *✍️ Drafted by Claude Code*
1 parent cfc2d28 commit 6aa8ead

9 files changed

Lines changed: 856 additions & 4 deletions

File tree

‎docs/02_concepts/07_convenience_methods.mdx‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@ The Apify client provides several convenience methods to handle actions that the
1818
- <ApiLink to="class/ActorClient#call">`ActorClient.call`</ApiLink> - Starts an Actor and waits for it to finish, handling network timeouts internally. Waits indefinitely by default, or up to the specified `wait_duration`.
1919
- <ApiLink to="class/ActorClient#start">`ActorClient.start`</ApiLink> - Starts an Actor and immediately returns the Run object without waiting for it to finish.
2020
- <ApiLink to="class/RunClient#wait_for_finish">`RunClient.wait_for_finish`</ApiLink> - Waits for an already-started run to reach a terminal status.
21+
- <ApiLink to="class/RunClient#iterate_dataset_items">`RunClient.iterate_dataset_items`</ApiLink> - Yields the items of the run's default dataset as the run pushes them, and returns once the run has finished and every item is read.
2122

2223
Additionally, storage-related resources offer flexible options for data retrieval:
2324

‎docs/02_concepts/08_pagination.mdx‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -83,3 +83,5 @@ The next example uses `iterate_items` on a dataset client to stream items past a
8383
</CodeBlock>
8484
</TabItem>
8585
</Tabs>
86+
87+
To read the items of a run that's still going, use <ApiLink to="class/RunClient#iterate_dataset_items">`RunClient.iterate_dataset_items`</ApiLink>. Its `offset` and `limit` also count dataset rows. For details, see [Retrieve Actor data](/api/client/python/docs/guides/retrieve-actor-data#read-items-while-the-run-is-going).

‎docs/03_guides/03_retrieve_actor_data.mdx‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,10 @@ import Tabs from '@theme/Tabs';
88
import TabItem from '@theme/TabItem';
99
import CodeBlock from '@theme/CodeBlock';
1010

11+
import ApiLink from '@theme/ApiLink';
12+
13+
import LiveItemsAsyncExample from '!!raw-loader!./code/03_live_items_async.py';
14+
import LiveItemsSyncExample from '!!raw-loader!./code/03_live_items_sync.py';
1115
import RetrieveAsyncExample from '!!raw-loader!./code/03_retrieve_async.py';
1216
import RetrieveSyncExample from '!!raw-loader!./code/03_retrieve_sync.py';
1317

@@ -27,3 +31,29 @@ The following example shows how to fetch datasets from an Actor's runs, paginate
2731
</CodeBlock>
2832
</TabItem>
2933
</Tabs>
34+
35+
## Read items while the run is going
36+
37+
To process a run's output before the run finishes, use <ApiLink to="class/RunClient#iterate_dataset_items">`RunClient.iterate_dataset_items`</ApiLink>. It yields the items of the run's default dataset shortly after the run pushes them, and it returns once the run has finished and every item is read.
38+
39+
Note that:
40+
41+
- Between polls, the iterator waits up to `poll_interval` (5 seconds by default) for the run to finish. Once it finishes, the iterator reads the remaining items right away. An explicit `timeout` has to leave room for that wait.
42+
- The item options are the same as in <ApiLink to="class/DatasetClient#iterate_items">`DatasetClient.iterate_items`</ApiLink>, except for `desc` and `signature`.
43+
- `offset` and `limit` count dataset rows, not the items you get back. With `clean`, `skip_empty` or `unwind`, the iterator can yield fewer or more items than `limit`.
44+
- A run that's `ABORTING` or `TIMING-OUT` can still push items, so the iterator keeps polling until the run reaches a terminal status.
45+
46+
The following example starts an Actor and prints its items as the run produces them:
47+
48+
<Tabs>
49+
<TabItem value="AsyncExample" label="Async client" default>
50+
<CodeBlock className="language-python">
51+
{LiveItemsAsyncExample}
52+
</CodeBlock>
53+
</TabItem>
54+
<TabItem value="SyncExample" label="Sync client">
55+
<CodeBlock className="language-python">
56+
{LiveItemsSyncExample}
57+
</CodeBlock>
58+
</TabItem>
59+
</Tabs>
Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
import asyncio
2+
3+
from apify_client import ApifyClientAsync
4+
5+
TOKEN = 'MY-APIFY-TOKEN'
6+
7+
8+
async def main() -> None:
9+
apify_client = ApifyClientAsync(TOKEN)
10+
11+
# Start the Actor without waiting for it to finish
12+
actor_client = apify_client.actor('username/actor-name')
13+
run = await actor_client.start(run_input={'query': 'web scraping'})
14+
15+
# Each item arrives shortly after the run pushes it. The loop ends once the run
16+
# has finished and every item is read.
17+
async for item in apify_client.run(run.id).iterate_dataset_items(skip_empty=True):
18+
print(item)
19+
20+
21+
if __name__ == '__main__':
22+
asyncio.run(main())
Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
from apify_client import ApifyClient
2+
3+
TOKEN = 'MY-APIFY-TOKEN'
4+
5+
6+
def main() -> None:
7+
apify_client = ApifyClient(TOKEN)
8+
9+
# Start the Actor without waiting for it to finish
10+
actor_client = apify_client.actor('username/actor-name')
11+
run = actor_client.start(run_input={'query': 'web scraping'})
12+
13+
# Each item arrives shortly after the run pushes it. The loop ends once the run
14+
# has finished and every item is read.
15+
for item in apify_client.run(run.id).iterate_dataset_items(skip_empty=True):
16+
print(item)
17+
18+
19+
if __name__ == '__main__':
20+
main()

‎src/apify_client/_resource_clients/_resource_client.py‎

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -42,6 +42,7 @@ def __init__(
4242
client_registry: Any,
4343
resource_id: str | None = None,
4444
params: dict | None = None,
45+
api_base_url: str | None = None,
4546
) -> None:
4647
"""Initialize the resource client.
4748
@@ -53,11 +54,13 @@ def __init__(
5354
client_registry: Bundle of client classes for dependency injection.
5455
resource_id: Optional resource ID for single-resource clients.
5556
params: Optional default parameters for all requests.
57+
api_base_url: Base URL of the API itself, for clients of top-level resources. Defaults to `base_url`.
5658
"""
5759
if resource_path.endswith('/'):
5860
raise ValueError('resource_path must not end with "/"')
5961

6062
self._base_url = base_url
63+
self._api_base_url = api_base_url or base_url
6164
self._public_base_url = public_base_url
6265
self._http_client = http_client
6366
self._default_params = params or {}
@@ -82,11 +85,12 @@ def _resource_url(self) -> str:
8285
def _base_client_kwargs(self) -> dict[str, Any]:
8386
"""Base kwargs for creating nested/child clients.
8487
85-
Returns dict with base_url, public_base_url, http_client, and client_registry. Caller adds
88+
Returns dict with base_url, api_base_url, public_base_url, http_client, and client_registry. Caller adds
8689
resource_path, resource_id, and params as needed.
8790
"""
8891
return {
8992
'base_url': self._resource_url,
93+
'api_base_url': self._api_base_url,
9094
'public_base_url': self._public_base_url,
9195
'http_client': self._http_client,
9296
'client_registry': self._client_registry,
@@ -197,6 +201,7 @@ def __init__(
197201
client_registry: ClientRegistry,
198202
resource_id: str | None = None,
199203
params: dict | None = None,
204+
api_base_url: str | None = None,
200205
) -> None:
201206
"""Initialize the resource client.
202207
@@ -208,6 +213,7 @@ def __init__(
208213
client_registry: Bundle of client classes for dependency injection.
209214
resource_id: Optional resource ID for single-resource clients.
210215
params: Optional default parameters for all requests.
216+
api_base_url: Base URL of the API itself, for clients of top-level resources. Defaults to `base_url`.
211217
"""
212218
super().__init__(
213219
base_url=base_url,
@@ -217,6 +223,7 @@ def __init__(
217223
client_registry=client_registry,
218224
resource_id=resource_id,
219225
params=params,
226+
api_base_url=api_base_url,
220227
)
221228

222229
def _get(self, *, timeout: Timeout) -> dict | None:
@@ -389,6 +396,7 @@ def __init__(
389396
client_registry: ClientRegistryAsync,
390397
resource_id: str | None = None,
391398
params: dict | None = None,
399+
api_base_url: str | None = None,
392400
) -> None:
393401
"""Initialize the resource client.
394402
@@ -400,6 +408,7 @@ def __init__(
400408
client_registry: Bundle of client classes for dependency injection.
401409
resource_id: Optional resource ID for single-resource clients.
402410
params: Optional default parameters for all requests.
411+
api_base_url: Base URL of the API itself, for clients of top-level resources. Defaults to `base_url`.
403412
"""
404413
super().__init__(
405414
base_url=base_url,
@@ -409,6 +418,7 @@ def __init__(
409418
client_registry=client_registry,
410419
resource_id=resource_id,
411420
params=params,
421+
api_base_url=api_base_url,
412422
)
413423

414424
async def _get(self, *, timeout: Timeout) -> dict | None:

0 commit comments

Comments
 (0)