Repository navigation
Commit 08747f7
committed
refactor: use asyncio.TaskGroup in the error snapshotter (#2321)
Converts the error snapshotter's concurrent HTML and screenshot saves to
`asyncio.TaskGroup`. Its caller catches and logs any `Exception`, so the
`ExceptionGroup` doesn't change behavior.
The other places from the issue keep their current code:
- `TaskGroup` wraps failures in an `ExceptionGroup`, which defeats the
crawler's type-based error handling (retries, error statistics) and
users' `except` clauses. That rules out the robots.txt skips,
`BrowserPool.new_page_with_each_plugin` and `infinite_scroll`, where the
gain is small.
- `ThrottlingRequestManager` reads are side-effect-free, so a task group
adds code without changing anything. `drop`, `purge` and the sub-manager
opening must let every call finish.
- `LocalEventManager` runs its readings in `to_thread`, which
cancellation can't stop.
- The autoscaled pool would start cancelling healthy workers when one
fails.
- The event manager, RQ background batches, `RecurringTask` and the
sitemap loader own tasks that outlive the call that starts them.
`_merge_async_generators` is an async generator.
Closes: #1722
*✍️ Drafted by Claude Code*1 parent 0bf67f5 commit 08747f7
1 file changed
Lines changed: 5 additions & 11 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
44 | 44 | | |
45 | 45 | | |
46 | 46 | | |
47 | | - | |
48 | 47 | | |
49 | | - | |
50 | | - | |
51 | | - | |
52 | | - | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
53 | 51 | | |
54 | | - | |
55 | | - | |
56 | | - | |
57 | | - | |
58 | | - | |
59 | | - | |
| 52 | + | |
| 53 | + | |
60 | 54 | | |
61 | 55 | | |
62 | 56 | | |
| |||
0 commit comments