You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 6aa8ead
Browse filesBrowse the repository at this point in the historyBrowse files
feat: Add live iteration over a run's dataset items (#1079)
Adds `RunClient.iterate_dataset_items()` and its async twin. They yield
a run's dataset items while the run is still pushing them and return
once the run has finished and the dataset is drained.
Each poll reads the dataset's `itemCount` and fetches pages whose
`limit` ends at it. The endpoint scans exactly `limit` rows, so offsets
stay exact even when `clean`, `skip_empty` or `unwind` change how many
items a page returns. Between polls, `wait_for_finish()` waits up to
`poll_interval` for the run to finish.
`itemCount` lags about 5 s, so after the run finishes, the rows past it
are read page by page until a page comes back empty. With `clean`,
`skip_empty` or `unwind`, one unfiltered `limit=1` read confirms the
end. Polling goes on through `ABORTING` and `TIMING-OUT`, since such a
run can still push items.
The arguments follow `DatasetClient.iterate_items` without `desc` and
`signature`, plus `poll_interval`.
Docs: a new section in the Retrieve Actor data guide, and a mention on
the convenience methods and pagination pages.
Closes: #1065
*✍️ Drafted by Claude Code*
Copy file name to clipboardExpand all lines: docs/02_concepts/07_convenience_methods.mdx
+1Lines changed: 1 addition & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -18,6 +18,7 @@ The Apify client provides several convenience methods to handle actions that the
18
18
- <ApiLinkto="class/ActorClient#call">`ActorClient.call`</ApiLink> - Starts an Actor and waits for it to finish, handling network timeouts internally. Waits indefinitely by default, or up to the specified `wait_duration`.
19
19
- <ApiLinkto="class/ActorClient#start">`ActorClient.start`</ApiLink> - Starts an Actor and immediately returns the Run object without waiting for it to finish.
20
20
- <ApiLinkto="class/RunClient#wait_for_finish">`RunClient.wait_for_finish`</ApiLink> - Waits for an already-started run to reach a terminal status.
21
+
- <ApiLinkto="class/RunClient#iterate_dataset_items">`RunClient.iterate_dataset_items`</ApiLink> - Yields the items of the run's default dataset as the run pushes them, and returns once the run has finished and every item is read.
21
22
22
23
Additionally, storage-related resources offer flexible options for data retrieval:
Copy file name to clipboardExpand all lines: docs/02_concepts/08_pagination.mdx
+2Lines changed: 2 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -83,3 +83,5 @@ The next example uses `iterate_items` on a dataset client to stream items past a
83
83
</CodeBlock>
84
84
</TabItem>
85
85
</Tabs>
86
+
87
+
To read the items of a run that's still going, use <ApiLinkto="class/RunClient#iterate_dataset_items">`RunClient.iterate_dataset_items`</ApiLink>. Its `offset` and `limit` also count dataset rows. For details, see [Retrieve Actor data](/api/client/python/docs/guides/retrieve-actor-data#read-items-while-the-run-is-going).
@@ -27,3 +31,29 @@ The following example shows how to fetch datasets from an Actor's runs, paginate
27
31
</CodeBlock>
28
32
</TabItem>
29
33
</Tabs>
34
+
35
+
## Read items while the run is going
36
+
37
+
To process a run's output before the run finishes, use <ApiLinkto="class/RunClient#iterate_dataset_items">`RunClient.iterate_dataset_items`</ApiLink>. It yields the items of the run's default dataset shortly after the run pushes them, and it returns once the run has finished and every item is read.
38
+
39
+
Note that:
40
+
41
+
- Between polls, the iterator waits up to `poll_interval` (5 seconds by default) for the run to finish. Once it finishes, the iterator reads the remaining items right away. An explicit `timeout` has to leave room for that wait.
42
+
- The item options are the same as in <ApiLinkto="class/DatasetClient#iterate_items">`DatasetClient.iterate_items`</ApiLink>, except for `desc` and `signature`.
43
+
-`offset` and `limit` count dataset rows, not the items you get back. With `clean`, `skip_empty` or `unwind`, the iterator can yield fewer or more items than `limit`.
44
+
- A run that's `ABORTING` or `TIMING-OUT` can still push items, so the iterator keeps polling until the run reaches a terminal status.
45
+
46
+
The following example starts an Actor and prints its items as the run produces them:
0 commit comments