You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit cc65186
Browse filesBrowse the repository at this point in the historyBrowse files
docs: fix links and the sidebar entry broken by the docs restructure (#2408)
Rebasing `v2` onto `master` combined `v2`'s changes with `master`'s docs
restructure (#2197). Four references ended up pointing at pages or ids
that no longer exist, and the Docusaurus build fails on them:
- `website/sidebars.js` lists `concepts/storage-clients`, which #2362
renamed to `storage-backends`. The build stops on the invalid sidebar
id.
- The Events concept page links `enum/Event`, but `Event` is a `StrEnum`
in v2 (#2295), so its API page is `class/Event`.
- The v1 upgrading guide links `../concepts/storage-clients`.
- The v2 upgrading guide links `../guides/proxy-management`,
`../guides/scaling-crawlers` and `../guides/request-loaders`, which now
live under `../concepts/`. The anchors they point at exist on the
concept pages.
- The Events and Configuration concept pages say that `Snapshotter`
drives autoscaling and link its API page. In v2, `Snapshotter` is
private and has no API page. The CPU and memory load signals listen to
the system info events, and the crawler builds its default
`ConcurrencySystem` from its configuration. The pages now say that and
link those classes. The site build doesn't check API links, so it didn't
flag these.
PR CI doesn't build the site, so only the docs release would catch
these. With this change, `pnpm build` passes locally. The remaining
broken-anchor warnings come from the frozen 0.6 API reference and from a
`Request#user_data` link that `master` has too.
*✍️ Drafted by Claude Code*
Crawlee components talk to each other through events. An event manager emits events such as "save your state now" or "here's the current CPU and memory usage", and any component can register a listener for them. The crawler doesn't need to know which components care about an event. The <ApiLinkto="class/Snapshotter">`Snapshotter`</ApiLink> listens to system info events to drive autoscaling, and statistics, storages, and your own code listen to persist state events to save their progress.
13
+
Crawlee components talk to each other through events. An event manager emits events such as "save your state now" or "here's the current CPU and memory usage", and any component can register a listener for them. The crawler doesn't need to know which components care about an event. The <ApiLinkto="class/CpuLoadSignal">`CpuLoadSignal`</ApiLink> and <ApiLinkto="class/MemoryLoadSignal">`MemoryLoadSignal`</ApiLink> listen to system info events to drive autoscaling, and statistics, storages, and your own code listen to persist state events to save their progress.
14
14
15
15
## Event manager
16
16
@@ -27,7 +27,7 @@ The event manager is an async context manager. The crawler enters it when `run()
27
27
28
28
## Events
29
29
30
-
The <ApiLinkto="enum/Event">`Event`</ApiLink> enum lists the event names. Each event comes with a payload model:
30
+
The <ApiLinkto="class/Event">`Event`</ApiLink> enum lists the event names. Each event comes with a payload model:
31
31
32
32
-`PERSIST_STATE` carries <ApiLinkto="class/EventPersistStateData">`EventPersistStateData`</ApiLink>. It's emitted when the event manager starts, then at every `persist_state_interval`, and one last time when it closes. Its `is_migrating` flag tells whether the process is about to be migrated.
33
33
-`SYSTEM_INFO` carries <ApiLinkto="class/EventSystemInfoData">`EventSystemInfoData`</ApiLink> with the CPU usage ratio and the memory used by the current process and its children. The `LocalEventManager` emits it at every `system_info_interval`.
Copy file name to clipboardExpand all lines: docs/03_concepts/24_configuration.mdx
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -70,7 +70,7 @@ If nothing is registered, the first call to `service_locator.get_configuration()
70
70
71
71
## Applying a configuration to one crawler
72
72
73
-
Pass a `Configuration` to the crawler's `configuration` argument to use it for that crawler only. The crawler keeps it in a service locator of its own and uses the configuration for the storages it opens, such as its default request queue and dataset, for the <ApiLinkto="class/Snapshotter">`Snapshotter`</ApiLink> that drives autoscaling, and for `internal_timeout`.
73
+
Pass a `Configuration` to the crawler's `configuration` argument to use it for that crawler only. The crawler keeps it in a service locator of its own and uses the configuration for the storages it opens, such as its default request queue and dataset, for the default <ApiLinkto="class/ConcurrencySystem">`ConcurrencySystem`</ApiLink> that drives autoscaling, and for `internal_timeout`.
Unlike tiers, this function doesn't track blocking per domain, so every request starts again from the first proxy. For details, see [Switching proxies on retries](../guides/proxy-management#switching-proxies-on-retries).
120
+
Unlike tiers, this function doesn't track blocking per domain, so every request starts again from the first proxy. For details, see [Switching proxies on retries](../concepts/proxy-management#switching-proxies-on-retries).
121
121
122
122
## `Request.forefront` is removed
123
123
@@ -196,7 +196,7 @@ The autoscaling classes now live in the public `crawlee.autoscaling` package, wh
196
196
-`run_task_function` receives the id the task is booked under.
197
197
-`current_concurrency` comes from the concurrency system, so a shared system may count the tasks of other pools too. The pool's own tasks are counted by the new `own_concurrency`.
198
198
-`is_finished_function` is called only when no new task can start and none of the pool's own tasks are in flight, and never while the pool is paused. In v1, it was called on every loop iteration, so a paused pool could finish. A pool whose `is_task_ready_function` keeps resolving to `True` doesn't finish, so make it resolve to `False` once the work runs out.
199
-
- The `cpu_overload_threshold`, `memory_overload_threshold`, `event_loop_overload_threshold` and `client_overload_threshold` options of `SystemStatus` are now the `overloaded_ratio` of the matching signal options in `load_signals`, such as `CpuLoadSignalOptions`. Its `max_snapshot_age` is now the `current_history` of the `ConcurrencySystem`. For details, see [Load signals](../guides/scaling-crawlers#load-signals).
199
+
- The `cpu_overload_threshold`, `memory_overload_threshold`, `event_loop_overload_threshold` and `client_overload_threshold` options of `SystemStatus` are now the `overloaded_ratio` of the matching signal options in `load_signals`, such as `CpuLoadSignalOptions`. Its `max_snapshot_age` is now the `current_history` of the `ConcurrencySystem`. For details, see [Load signals](../concepts/scaling-crawlers#load-signals).
200
200
-`pause()` is a coroutine that returns once the pool's tasks in progress finish. With the new `timeout` option, it raises `TimeoutError` if they take longer, and the pool stays paused. Await it, since a call without `await` doesn't pause the pool.
201
201
202
202
For example:
@@ -233,13 +233,13 @@ async with ConcurrencySystem(settings) as concurrency_system:
233
233
234
234
## `max_tasks_per_minute` counts the last minute
235
235
236
-
`ConcurrencySettings.max_tasks_per_minute` now caps the tasks started over the last 60 seconds, the same as Crawlee JS. In v1, the pool spaced task starts evenly, `60 / max_tasks_per_minute` seconds apart. In v2, tasks start as fast as the concurrency allows until the cap is reached. The next ones wait until the oldest starts age out of the window. The rate over a minute stays the same, but tasks start in bursts. For details, see [Max tasks per minute](../guides/scaling-crawlers#max-tasks-per-minute).
236
+
`ConcurrencySettings.max_tasks_per_minute` now caps the tasks started over the last 60 seconds, the same as Crawlee JS. In v1, the pool spaced task starts evenly, `60 / max_tasks_per_minute` seconds apart. In v2, tasks start as fast as the concurrency allows until the cap is reached. The next ones wait until the oldest starts age out of the window. The rate over a minute stays the same, but tasks start in bursts. For details, see [Max tasks per minute](../concepts/scaling-crawlers#max-tasks-per-minute).
237
237
238
238
The value has to be a whole number of at least 1, or infinity, which is the default. A fractional value, such as `0.5` or `7.5`, now raises `ValueError`.
239
239
240
240
## `SystemInfo.client_info` is renamed
241
241
242
-
`SystemInfo.client_info` is renamed to `storage_backend_info`, after the `StorageBackendLoadSignal` that reports it. The new `load_signal_info` holds the verdicts of custom load signals by name. For how to configure the signals or add your own, see [Load signals](../guides/scaling-crawlers#load-signals).
242
+
`SystemInfo.client_info` is renamed to `storage_backend_info`, after the `StorageBackendLoadSignal` that reports it. The new `load_signal_info` holds the verdicts of custom load signals by name. For how to configure the signals or add your own, see [Load signals](../concepts/scaling-crawlers#load-signals).
If you have a custom request manager, implement `record_pacing_signal`. Return `False` if the manager doesn't pace requests or doesn't handle the signal's `reason`, and forward every signal if it wraps another manager. For details, see [Pacing signals](../guides/request-loaders#pacing-signals).
344
+
If you have a custom request manager, implement `record_pacing_signal`. Return `False` if the manager doesn't pace requests or doesn't handle the signal's `reason`, and forward every signal if it wraps another manager. For details, see [Pacing signals](../concepts/request-loaders#pacing-signals).
345
345
346
346
`ThrottlingRequestManager` now rejects a `base_delay` or `max_delay` that isn't positive, so a zero `base_delay` no longer works as a way to honor only `Retry-After`.
0 commit comments