@@ -372,13 +372,10 @@ def web_scrape_sitemap(
372372 timeout : float | httpx .Timeout | None | NotGiven = not_given ,
373373 ) -> WebWebScrapeSitemapResponse :
374374 """
375- Crawls the sitemap of the given domain and returns all discovered page URLs.
376- Supports sitemap index files (recursive), parallel fetching with concurrency
377- control, deduplication, and filters out non-page resources (images, PDFs, etc.).
375+ Crawl an entire website's sitemap and return all discovered page URLs
378376
379377 Args:
380- domain: Domain name to crawl sitemaps for (e.g., 'example.com'). The domain will be
381- automatically normalized and validated.
378+ domain: Domain to build a sitemap for
382379
383380 max_links: Maximum number of links to return from the sitemap crawl. Defaults to 10,000.
384381 Minimum is 1, maximum is 100,000.
@@ -747,13 +744,10 @@ async def web_scrape_sitemap(
747744 timeout : float | httpx .Timeout | None | NotGiven = not_given ,
748745 ) -> WebWebScrapeSitemapResponse :
749746 """
750- Crawls the sitemap of the given domain and returns all discovered page URLs.
751- Supports sitemap index files (recursive), parallel fetching with concurrency
752- control, deduplication, and filters out non-page resources (images, PDFs, etc.).
747+ Crawl an entire website's sitemap and return all discovered page URLs
753748
754749 Args:
755- domain: Domain name to crawl sitemaps for (e.g., 'example.com'). The domain will be
756- automatically normalized and validated.
750+ domain: Domain to build a sitemap for
757751
758752 max_links: Maximum number of links to return from the sitemap crawl. Defaults to 10,000.
759753 Minimum is 1, maximum is 100,000.
0 commit comments