Skip to content

Commit 3dd8ca5

Browse files
authored
feat: retry starting a run on memory and concurrent-runs limits (#1081)
Adds a `wait_for_resources` argument to `ActorClient.start`/`call` and `TaskClient.start`/`call`, sync and async. When set, a start that the API rejects with `actor-memory-limit-exceeded` or `concurrent-runs-limit-exceeded` is retried every 10 seconds. Orchestrators no longer need their own wait-and-retry loop. - `True` retries until the run starts. A `timedelta` limits the retrying, after which the last 402 error is raised. - Any other error is raised right away. - In `call`, the retry time doesn't count toward `wait_duration`. - A file-like input is read into memory before the first attempt, so every retry sends it whole. - A run that asks for more memory than the account's whole memory limit gets the same `actor-memory-limit-exceeded` error and never starts, so `True` retries it forever. The API has no separate error type for that case. The docs call it out. The Retries concept page gets a section with sync and async examples. The JS twin is apify/apify-client-js#1091 and uses the same design. Closes: #1071 *✍️ Drafted by Claude Code*
1 parent ae17148 commit 3dd8ca5

7 files changed

Lines changed: 524 additions & 28 deletions

File tree

‎docs/02_concepts/05_retries.mdx‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,8 @@ import ApiLink from '@theme/ApiLink';
1212

1313
import RetriesAsyncExample from '!!raw-loader!./code/05_retries_async.py';
1414
import RetriesSyncExample from '!!raw-loader!./code/05_retries_sync.py';
15+
import WaitForResourcesAsyncExample from '!!raw-loader!./code/05_wait_for_resources_async.py';
16+
import WaitForResourcesSyncExample from '!!raw-loader!./code/05_wait_for_resources_sync.py';
1517

1618
The Apify client automatically retries requests that fail due to:
1719

@@ -43,3 +45,27 @@ Retries with exponential backoff help reduce the load on the server and increase
4345
</CodeBlock>
4446
</TabItem>
4547
</Tabs>
48+
49+
## Wait for resources to start a run
50+
51+
Starting a run fails with an HTTP 402 error when the account doesn't have enough free memory for the run, or when it already runs as many Actors as its plan allows. The error `type` is `actor-memory-limit-exceeded` or `concurrent-runs-limit-exceeded`. The client doesn't retry these errors on its own, since they clear only after other runs or builds of the account finish.
52+
53+
To keep retrying the start until the resources free up, set the `wait_for_resources` argument of <ApiLink to="class/ActorClient#start">`ActorClient.start`</ApiLink>, <ApiLink to="class/ActorClient#call">`ActorClient.call`</ApiLink>, or the same methods of <ApiLink to="class/TaskClient">`TaskClient`</ApiLink>. The client then retries the start every 10 seconds, and the argument value sets how long:
54+
55+
- `True` retries until the run starts.
56+
- A `timedelta` stops retrying after that time and raises the last error.
57+
58+
Any other error raises right away. A run that asks for more memory than the account's whole memory limit gets the same `actor-memory-limit-exceeded` error and never starts, so with `True` the client retries it forever. In `call`, the time spent retrying doesn't count toward `wait_duration`.
59+
60+
<Tabs>
61+
<TabItem value="AsyncExample" label="Async client" default>
62+
<CodeBlock className="language-python">
63+
{WaitForResourcesAsyncExample}
64+
</CodeBlock>
65+
</TabItem>
66+
<TabItem value="SyncExample" label="Sync client">
67+
<CodeBlock className="language-python">
68+
{WaitForResourcesSyncExample}
69+
</CodeBlock>
70+
</TabItem>
71+
</Tabs>
Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
from datetime import timedelta
2+
3+
from apify_client import ApifyClientAsync
4+
5+
TOKEN = 'MY-APIFY-TOKEN'
6+
7+
8+
async def main() -> None:
9+
apify_client = ApifyClientAsync(TOKEN)
10+
11+
# Retry the start until the account has the resources for the run.
12+
run = await apify_client.actor('username/actor-name').call(wait_for_resources=True)
13+
14+
# Stop retrying after 10 minutes and raise the last error.
15+
started_run = await apify_client.task('username~task-name').start(
16+
wait_for_resources=timedelta(minutes=10),
17+
)
Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
from datetime import timedelta
2+
3+
from apify_client import ApifyClient
4+
5+
TOKEN = 'MY-APIFY-TOKEN'
6+
7+
8+
def main() -> None:
9+
apify_client = ApifyClient(TOKEN)
10+
11+
# Retry the start until the account has the resources for the run.
12+
run = apify_client.actor('username/actor-name').call(wait_for_resources=True)
13+
14+
# Stop retrying after 10 minutes and raise the last error.
15+
started_run = apify_client.task('username~task-name').start(
16+
wait_for_resources=timedelta(minutes=10),
17+
)

‎src/apify_client/_resource_clients/actor.py‎

Lines changed: 65 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,11 @@
2626
from apify_client._utils.encoding import encode_key_value_store_record_value, encode_webhooks_to_base64
2727
from apify_client._utils.http import response_to_dict
2828
from apify_client._utils.time import to_seconds
29+
from apify_client._utils.wait_for_resources import (
30+
prepare_resendable_body,
31+
start_waiting_for_resources,
32+
start_waiting_for_resources_async,
33+
)
2934

3035
if TYPE_CHECKING:
3136
from datetime import timedelta
@@ -229,6 +234,7 @@ def start(
229234
force_permission_level: ActorPermissionLevel | None = None,
230235
wait_for_finish: int | None = None,
231236
webhooks: WebhooksList | None = None,
237+
wait_for_resources: bool | timedelta = False,
232238
timeout: Timeout = 'medium',
233239
) -> Run:
234240
"""Start the Actor and immediately return the Run object.
@@ -263,12 +269,21 @@ def start(
263269
* `event_types`: List of `WebhookEventType` values which trigger the webhook.
264270
* `request_url`: URL to which to send the webhook HTTP request.
265271
* `payload_template`: Optional template for the request payload.
272+
wait_for_resources: Retry the start while the account lacks the memory or a concurrent-run slot for the run,
273+
that is while the API rejects it with an `ApifyApiError` of type `actor-memory-limit-exceeded` or
274+
`concurrent-runs-limit-exceeded`. Both clear as other runs or builds finish. The start is retried
275+
every 10 seconds, and any other error is raised right away. `True` retries until the run starts, a
276+
`timedelta` stops retrying after that long and raises the last error. A run that requests more memory
277+
than the whole memory limit of the account is rejected with `actor-memory-limit-exceeded` as well and
278+
never starts, so `True` retries it forever. A streamed `run_input` that cannot be rewound, such as a
279+
generator, is sent only once, so its start is not retried.
266280
timeout: Timeout for the API HTTP request.
267281
268282
Returns:
269283
The run object.
270284
"""
271285
run_input, content_type = encode_key_value_store_record_value(run_input, content_type=content_type)
286+
run_input, wait_for_resources = prepare_resendable_body(run_input, wait_for_resources=wait_for_resources)
272287

273288
request_params = self._build_params(
274289
build=build,
@@ -282,13 +297,16 @@ def start(
282297
webhooks=encode_webhooks_to_base64(webhooks),
283298
)
284299

285-
response = self._http_client.call(
286-
url=self._build_url('runs'),
287-
method='POST',
288-
headers={'content-type': content_type},
289-
data=run_input,
290-
params=request_params,
291-
timeout=timeout,
300+
response = start_waiting_for_resources(
301+
lambda: self._http_client.call(
302+
url=self._build_url('runs'),
303+
method='POST',
304+
headers={'content-type': content_type},
305+
data=run_input,
306+
params=request_params,
307+
timeout=timeout,
308+
),
309+
wait_for_resources=wait_for_resources,
292310
)
293311

294312
result = response_to_dict(response)
@@ -308,6 +326,7 @@ def call(
308326
webhooks: WebhooksList | None = None,
309327
force_permission_level: ActorPermissionLevel | None = None,
310328
wait_duration: timedelta | None = None,
329+
wait_for_resources: bool | timedelta = False,
311330
logger: Logger | Literal['default'] | None = 'default',
312331
timeout: Timeout = 'no_timeout',
313332
) -> Run | None:
@@ -341,6 +360,14 @@ def call(
341360
a webhook set up for the Actor, you do not have to add it again here.
342361
wait_duration: The maximum time the server waits for the run to finish. If not provided,
343362
waits indefinitely.
363+
wait_for_resources: Retry the start while the account lacks the memory or a concurrent-run slot for the run,
364+
that is while the API rejects it with an `ApifyApiError` of type `actor-memory-limit-exceeded` or
365+
`concurrent-runs-limit-exceeded`. Both clear as other runs or builds finish. The start is retried
366+
every 10 seconds, and any other error is raised right away. `True` retries until the run starts, a
367+
`timedelta` stops retrying after that long and raises the last error. A run that requests more memory
368+
than the whole memory limit of the account is rejected with `actor-memory-limit-exceeded` as well and
369+
never starts, so `True` retries it forever. The time spent retrying doesn't count toward
370+
`wait_duration`.
344371
logger: Logger used to redirect logs from the Actor run. Using "default" literal means that a predefined
345372
default logger will be used. Setting `None` will disable any log propagation. Passing custom logger
346373
will redirect logs to the provided logger. The logger is also used to capture status and status message
@@ -361,6 +388,7 @@ def call(
361388
run_timeout=run_timeout,
362389
webhooks=webhooks,
363390
force_permission_level=force_permission_level,
391+
wait_for_resources=wait_for_resources,
364392
timeout=timeout,
365393
)
366394
run_client = self._client_registry.run_client(
@@ -740,6 +768,7 @@ async def start(
740768
force_permission_level: ActorPermissionLevel | None = None,
741769
wait_for_finish: int | None = None,
742770
webhooks: WebhooksList | None = None,
771+
wait_for_resources: bool | timedelta = False,
743772
timeout: Timeout = 'medium',
744773
) -> Run:
745774
"""Start the Actor and immediately return the Run object.
@@ -774,12 +803,21 @@ async def start(
774803
* `event_types`: List of `WebhookEventType` values which trigger the webhook.
775804
* `request_url`: URL to which to send the webhook HTTP request.
776805
* `payload_template`: Optional template for the request payload.
806+
wait_for_resources: Retry the start while the account lacks the memory or a concurrent-run slot for the run,
807+
that is while the API rejects it with an `ApifyApiError` of type `actor-memory-limit-exceeded` or
808+
`concurrent-runs-limit-exceeded`. Both clear as other runs or builds finish. The start is retried
809+
every 10 seconds, and any other error is raised right away. `True` retries until the run starts, a
810+
`timedelta` stops retrying after that long and raises the last error. A run that requests more memory
811+
than the whole memory limit of the account is rejected with `actor-memory-limit-exceeded` as well and
812+
never starts, so `True` retries it forever. A streamed `run_input` that cannot be rewound, such as a
813+
generator, is sent only once, so its start is not retried.
777814
timeout: Timeout for the API HTTP request.
778815
779816
Returns:
780817
The run object.
781818
"""
782819
run_input, content_type = encode_key_value_store_record_value(run_input, content_type=content_type)
820+
run_input, wait_for_resources = prepare_resendable_body(run_input, wait_for_resources=wait_for_resources)
783821

784822
request_params = self._build_params(
785823
build=build,
@@ -793,13 +831,16 @@ async def start(
793831
webhooks=encode_webhooks_to_base64(webhooks),
794832
)
795833

796-
response = await self._http_client.call(
797-
url=self._build_url('runs'),
798-
method='POST',
799-
headers={'content-type': content_type},
800-
data=run_input,
801-
params=request_params,
802-
timeout=timeout,
834+
response = await start_waiting_for_resources_async(
835+
lambda: self._http_client.call(
836+
url=self._build_url('runs'),
837+
method='POST',
838+
headers={'content-type': content_type},
839+
data=run_input,
840+
params=request_params,
841+
timeout=timeout,
842+
),
843+
wait_for_resources=wait_for_resources,
803844
)
804845

805846
result = response_to_dict(response)
@@ -819,6 +860,7 @@ async def call(
819860
webhooks: WebhooksList | None = None,
820861
force_permission_level: ActorPermissionLevel | None = None,
821862
wait_duration: timedelta | None = None,
863+
wait_for_resources: bool | timedelta = False,
822864
logger: Logger | Literal['default'] | None = 'default',
823865
timeout: Timeout = 'no_timeout',
824866
) -> Run | None:
@@ -852,6 +894,14 @@ async def call(
852894
a webhook set up for the Actor, you do not have to add it again here.
853895
wait_duration: The maximum time the server waits for the run to finish. If not provided,
854896
waits indefinitely.
897+
wait_for_resources: Retry the start while the account lacks the memory or a concurrent-run slot for the run,
898+
that is while the API rejects it with an `ApifyApiError` of type `actor-memory-limit-exceeded` or
899+
`concurrent-runs-limit-exceeded`. Both clear as other runs or builds finish. The start is retried
900+
every 10 seconds, and any other error is raised right away. `True` retries until the run starts, a
901+
`timedelta` stops retrying after that long and raises the last error. A run that requests more memory
902+
than the whole memory limit of the account is rejected with `actor-memory-limit-exceeded` as well and
903+
never starts, so `True` retries it forever. The time spent retrying doesn't count toward
904+
`wait_duration`.
855905
logger: Logger used to redirect logs from the Actor run. Using "default" literal means that a predefined
856906
default logger will be used. Setting `None` will disable any log propagation. Passing custom logger
857907
will redirect logs to the provided logger. The logger is also used to capture status and status message
@@ -872,6 +922,7 @@ async def call(
872922
run_timeout=run_timeout,
873923
webhooks=webhooks,
874924
force_permission_level=force_permission_level,
925+
wait_for_resources=wait_for_resources,
875926
timeout=timeout,
876927
)
877928

0 commit comments

Comments
 (0)