Skip to content

Commit 08747f7

Browse files
committed
refactor: use asyncio.TaskGroup in the error snapshotter (#2321)
Converts the error snapshotter's concurrent HTML and screenshot saves to `asyncio.TaskGroup`. Its caller catches and logs any `Exception`, so the `ExceptionGroup` doesn't change behavior. The other places from the issue keep their current code: - `TaskGroup` wraps failures in an `ExceptionGroup`, which defeats the crawler's type-based error handling (retries, error statistics) and users' `except` clauses. That rules out the robots.txt skips, `BrowserPool.new_page_with_each_plugin` and `infinite_scroll`, where the gain is small. - `ThrottlingRequestManager` reads are side-effect-free, so a task group adds code without changing anything. `drop`, `purge` and the sub-manager opening must let every call finish. - `LocalEventManager` runs its readings in `to_thread`, which cancellation can't stop. - The autoscaled pool would start cancelling healthy workers when one fails. - The event manager, RQ background batches, `RecurringTask` and the sitemap loader own tasks that outlive the call that starts them. `_merge_async_generators` is an async generator. Closes: #1722 *✍️ Drafted by Claude Code*
1 parent 0bf67f5 commit 08747f7

1 file changed

Lines changed: 5 additions & 11 deletions

File tree

‎src/crawlee/statistics/_error_snapshotter.py‎

Lines changed: 5 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -44,19 +44,13 @@ async def capture_snapshot(
4444
if snapshot := await context.get_snapshot():
4545
kvs = await KeyValueStore.open(name=self._kvs_name)
4646
snapshot_base_name = self._get_snapshot_base_name(error_message, file_and_line)
47-
snapshot_save_tasks = list[asyncio.Task]()
4847

49-
if snapshot.html:
50-
snapshot_save_tasks.append(
51-
asyncio.create_task(self._save_html(kvs, snapshot.html, base_name=snapshot_base_name))
52-
)
48+
async with asyncio.TaskGroup() as tg:
49+
if snapshot.html:
50+
tg.create_task(self._save_html(kvs, snapshot.html, base_name=snapshot_base_name))
5351

54-
if snapshot.screenshot:
55-
snapshot_save_tasks.append(
56-
asyncio.create_task(self._save_screenshot(kvs, snapshot.screenshot, base_name=snapshot_base_name))
57-
)
58-
59-
await asyncio.gather(*snapshot_save_tasks)
52+
if snapshot.screenshot:
53+
tg.create_task(self._save_screenshot(kvs, snapshot.screenshot, base_name=snapshot_base_name))
6054

6155
async def _save_html(self, kvs: KeyValueStore, html: str, base_name: str) -> None:
6256
file_name = f'{base_name}.html'

0 commit comments

Comments
 (0)