You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: home/concepts/scrape-vs-crawl.mdx
+2-2Lines changed: 2 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -84,9 +84,9 @@ A rule of thumb:
84
84
A crawl result and a batch result are the same thing as far as your code is concerned: a job with a `results` array where each entry is one page. You can write one handler and point it at either.
85
85
86
86
```ts
87
-
const result =awaitclient.read({ urls: [...] }); // batch
87
+
const result =awaitreader.read({ urls: [...] }); // batch
88
88
// or
89
-
const result =awaitclient.read({ url, maxPages: 100 }); // crawl
89
+
const result =awaitreader.read({ url, maxPages: 100 }); // crawl
@@ -49,7 +49,7 @@ If your scrapes are failing with `upstream_unavailable`, `scrape_timeout`, or ar
49
49
When you know ahead of time a target is hostile (Amazon, LinkedIn, booking.com, ticketing sites, some news aggregators), skip the optimistic `auto` attempt and go straight to stealth:
Copy file name to clipboardExpand all lines: home/guides/advanced/batch-scraping.mdx
+5-5Lines changed: 5 additions & 5 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -8,7 +8,7 @@ When you have a list of URLs (from a sitemap, an RSS feed, a search result, or y
8
8
## The basic request
9
9
10
10
```ts
11
-
const result =awaitclient.read({
11
+
const result =awaitreader.read({
12
12
urls: [
13
13
"https://example.com/article/1",
14
14
"https://example.com/article/2",
@@ -36,7 +36,7 @@ A loop of sync scrapes eats your rate limit, your connection pool, and your pati
36
36
```ts
37
37
// One request per URL: don't do this
38
38
for (const url ofurls) {
39
-
awaitclient.read({ url });
39
+
awaitreader.read({ url });
40
40
}
41
41
```
42
42
@@ -47,7 +47,7 @@ The batch version is **one** API call. Reader handles parallelism internally and
47
47
By default Reader picks a sensible parallelism level for your batch. For very large batches or target sites you want to be gentle with, set `batchConcurrency` explicitly:
48
48
49
49
```ts
50
-
awaitclient.read({
50
+
awaitreader.read({
51
51
urls: manyUrls,
52
52
batchConcurrency: 5, // Reader runs up to 5 scrapes in parallel
0 commit comments