Skip to content

Commit cc65186

Browse files
authored
docs: fix links and the sidebar entry broken by the docs restructure (#2408)
Rebasing `v2` onto `master` combined `v2`'s changes with `master`'s docs restructure (#2197). Four references ended up pointing at pages or ids that no longer exist, and the Docusaurus build fails on them: - `website/sidebars.js` lists `concepts/storage-clients`, which #2362 renamed to `storage-backends`. The build stops on the invalid sidebar id. - The Events concept page links `enum/Event`, but `Event` is a `StrEnum` in v2 (#2295), so its API page is `class/Event`. - The v1 upgrading guide links `../concepts/storage-clients`. - The v2 upgrading guide links `../guides/proxy-management`, `../guides/scaling-crawlers` and `../guides/request-loaders`, which now live under `../concepts/`. The anchors they point at exist on the concept pages. - The Events and Configuration concept pages say that `Snapshotter` drives autoscaling and link its API page. In v2, `Snapshotter` is private and has no API page. The CPU and memory load signals listen to the system info events, and the crawler builds its default `ConcurrencySystem` from its configuration. The pages now say that and link those classes. The site build doesn't check API links, so it didn't flag these. PR CI doesn't build the site, so only the docs release would catch these. With this change, `pnpm build` passes locally. The remaining broken-anchor warnings come from the frozen 0.6 API reference and from a `Request#user_data` link that `master` has too. *✍️ Drafted by Claude Code*
1 parent f46a104 commit cc65186

5 files changed

Lines changed: 10 additions & 10 deletions

File tree

‎docs/03_concepts/21_events.mdx‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@ import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
1010
import ListenToEventsExample from '!!raw-loader!roa-loader!./code_examples/events/listen_to_events.py';
1111
import UseStateExample from '!!raw-loader!roa-loader!./code_examples/events/use_state.py';
1212

13-
Crawlee components talk to each other through events. An event manager emits events such as "save your state now" or "here's the current CPU and memory usage", and any component can register a listener for them. The crawler doesn't need to know which components care about an event. The <ApiLink to="class/Snapshotter">`Snapshotter`</ApiLink> listens to system info events to drive autoscaling, and statistics, storages, and your own code listen to persist state events to save their progress.
13+
Crawlee components talk to each other through events. An event manager emits events such as "save your state now" or "here's the current CPU and memory usage", and any component can register a listener for them. The crawler doesn't need to know which components care about an event. The <ApiLink to="class/CpuLoadSignal">`CpuLoadSignal`</ApiLink> and <ApiLink to="class/MemoryLoadSignal">`MemoryLoadSignal`</ApiLink> listen to system info events to drive autoscaling, and statistics, storages, and your own code listen to persist state events to save their progress.
1414

1515
## Event manager
1616

@@ -27,7 +27,7 @@ The event manager is an async context manager. The crawler enters it when `run()
2727

2828
## Events
2929

30-
The <ApiLink to="enum/Event">`Event`</ApiLink> enum lists the event names. Each event comes with a payload model:
30+
The <ApiLink to="class/Event">`Event`</ApiLink> enum lists the event names. Each event comes with a payload model:
3131

3232
- `PERSIST_STATE` carries <ApiLink to="class/EventPersistStateData">`EventPersistStateData`</ApiLink>. It's emitted when the event manager starts, then at every `persist_state_interval`, and one last time when it closes. Its `is_migrating` flag tells whether the process is about to be migrated.
3333
- `SYSTEM_INFO` carries <ApiLink to="class/EventSystemInfoData">`EventSystemInfoData`</ApiLink> with the CPU usage ratio and the memory used by the current process and its children. The `LocalEventManager` emits it at every `system_info_interval`.

‎docs/03_concepts/24_configuration.mdx‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -70,7 +70,7 @@ If nothing is registered, the first call to `service_locator.get_configuration()
7070

7171
## Applying a configuration to one crawler
7272

73-
Pass a `Configuration` to the crawler's `configuration` argument to use it for that crawler only. The crawler keeps it in a service locator of its own and uses the configuration for the storages it opens, such as its default request queue and dataset, for the <ApiLink to="class/Snapshotter">`Snapshotter`</ApiLink> that drives autoscaling, and for `internal_timeout`.
73+
Pass a `Configuration` to the crawler's `configuration` argument to use it for that crawler only. The crawler keeps it in a service locator of its own and uses the configuration for the storages it opens, such as its default request queue and dataset, for the default <ApiLink to="class/ConcurrencySystem">`ConcurrencySystem`</ApiLink> that drives autoscaling, and for `internal_timeout`.
7474

7575
<RunnableCodeBlock className="language-python" language="python">
7676
{CrawlerConfigurationExample}

‎docs/06_upgrading/upgrading_to_v1.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -116,7 +116,7 @@ Some changes in the related model classes:
116116

117117
In v1.0, the storage client system has been completely reworked to simplify implementation and make custom storage clients easier to write.
118118

119-
See the [Storage clients](../concepts/storage-clients) page for more details.
119+
See the [Storage backends](../concepts/storage-backends) page for more details.
120120

121121
### New dedicated storage clients
122122

‎docs/06_upgrading/upgrading_to_v2.md‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -117,7 +117,7 @@ def pick_proxy_url(session_id=None, request=None):
117117
proxy_configuration = ProxyConfiguration(new_url_function=pick_proxy_url)
118118
```
119119

120-
Unlike tiers, this function doesn't track blocking per domain, so every request starts again from the first proxy. For details, see [Switching proxies on retries](../guides/proxy-management#switching-proxies-on-retries).
120+
Unlike tiers, this function doesn't track blocking per domain, so every request starts again from the first proxy. For details, see [Switching proxies on retries](../concepts/proxy-management#switching-proxies-on-retries).
121121

122122
## `Request.forefront` is removed
123123

@@ -196,7 +196,7 @@ The autoscaling classes now live in the public `crawlee.autoscaling` package, wh
196196
- `run_task_function` receives the id the task is booked under.
197197
- `current_concurrency` comes from the concurrency system, so a shared system may count the tasks of other pools too. The pool's own tasks are counted by the new `own_concurrency`.
198198
- `is_finished_function` is called only when no new task can start and none of the pool's own tasks are in flight, and never while the pool is paused. In v1, it was called on every loop iteration, so a paused pool could finish. A pool whose `is_task_ready_function` keeps resolving to `True` doesn't finish, so make it resolve to `False` once the work runs out.
199-
- The `cpu_overload_threshold`, `memory_overload_threshold`, `event_loop_overload_threshold` and `client_overload_threshold` options of `SystemStatus` are now the `overloaded_ratio` of the matching signal options in `load_signals`, such as `CpuLoadSignalOptions`. Its `max_snapshot_age` is now the `current_history` of the `ConcurrencySystem`. For details, see [Load signals](../guides/scaling-crawlers#load-signals).
199+
- The `cpu_overload_threshold`, `memory_overload_threshold`, `event_loop_overload_threshold` and `client_overload_threshold` options of `SystemStatus` are now the `overloaded_ratio` of the matching signal options in `load_signals`, such as `CpuLoadSignalOptions`. Its `max_snapshot_age` is now the `current_history` of the `ConcurrencySystem`. For details, see [Load signals](../concepts/scaling-crawlers#load-signals).
200200
- `pause()` is a coroutine that returns once the pool's tasks in progress finish. With the new `timeout` option, it raises `TimeoutError` if they take longer, and the pool stays paused. Await it, since a call without `await` doesn't pause the pool.
201201

202202
For example:
@@ -233,13 +233,13 @@ async with ConcurrencySystem(settings) as concurrency_system:
233233

234234
## `max_tasks_per_minute` counts the last minute
235235

236-
`ConcurrencySettings.max_tasks_per_minute` now caps the tasks started over the last 60 seconds, the same as Crawlee JS. In v1, the pool spaced task starts evenly, `60 / max_tasks_per_minute` seconds apart. In v2, tasks start as fast as the concurrency allows until the cap is reached. The next ones wait until the oldest starts age out of the window. The rate over a minute stays the same, but tasks start in bursts. For details, see [Max tasks per minute](../guides/scaling-crawlers#max-tasks-per-minute).
236+
`ConcurrencySettings.max_tasks_per_minute` now caps the tasks started over the last 60 seconds, the same as Crawlee JS. In v1, the pool spaced task starts evenly, `60 / max_tasks_per_minute` seconds apart. In v2, tasks start as fast as the concurrency allows until the cap is reached. The next ones wait until the oldest starts age out of the window. The rate over a minute stays the same, but tasks start in bursts. For details, see [Max tasks per minute](../concepts/scaling-crawlers#max-tasks-per-minute).
237237

238238
The value has to be a whole number of at least 1, or infinity, which is the default. A fractional value, such as `0.5` or `7.5`, now raises `ValueError`.
239239

240240
## `SystemInfo.client_info` is renamed
241241

242-
`SystemInfo.client_info` is renamed to `storage_backend_info`, after the `StorageBackendLoadSignal` that reports it. The new `load_signal_info` holds the verdicts of custom load signals by name. For how to configure the signals or add your own, see [Load signals](../guides/scaling-crawlers#load-signals).
242+
`SystemInfo.client_info` is renamed to `storage_backend_info`, after the `StorageBackendLoadSignal` that reports it. The new `load_signal_info` holds the verdicts of custom load signals by name. For how to configure the signals or add your own, see [Load signals](../concepts/scaling-crawlers#load-signals).
243243

244244
## `max_client_errors` defaults to 3
245245

@@ -341,7 +341,7 @@ throttler.record_pacing_signal(
341341
)
342342
```
343343

344-
If you have a custom request manager, implement `record_pacing_signal`. Return `False` if the manager doesn't pace requests or doesn't handle the signal's `reason`, and forward every signal if it wraps another manager. For details, see [Pacing signals](../guides/request-loaders#pacing-signals).
344+
If you have a custom request manager, implement `record_pacing_signal`. Return `False` if the manager doesn't pace requests or doesn't handle the signal's `reason`, and forward every signal if it wraps another manager. For details, see [Pacing signals](../concepts/request-loaders#pacing-signals).
345345

346346
`ThrottlingRequestManager` now rejects a `base_delay` or `max_delay` that isn't positive, so a zero `base_delay` no longer works as a way to honor only `Retry-After`.
347347

‎website/sidebars.js‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -72,7 +72,7 @@ module.exports = {
7272
'Storage',
7373
'storage',
7474
'Where crawlers keep data and requests: datasets, key-value stores, request queues, storage backends, and request loaders.',
75-
['storages', 'storage-clients', 'request-loaders'],
75+
['storages', 'storage-backends', 'request-loaders'],
7676
),
7777
conceptCategory(
7878
'HTTP and networking',

0 commit comments

Comments
 (0)