|
1 | 1 | --- |
2 | 2 | title: "Proxy modes" |
3 | | -description: "Three ways Reader can fetch a page: standard, stealth, and auto. Pick one, or let Reader decide." |
| 3 | +description: "Two ways Reader can fetch a page: standard and premium. Pick one, or use the default." |
4 | 4 | --- |
5 | 5 |
|
6 | 6 | Not every page is equally easy to fetch. A static blog post loads in a hundred milliseconds. A product detail page behind aggressive bot protection might refuse you three times before finally serving content. Reader handles both. You pick how. |
7 | 7 |
|
8 | | -## The three modes |
| 8 | +## The two modes |
9 | 9 |
|
10 | 10 | | Mode | What it's for | Cost | Typical latency | |
11 | 11 | | --- | --- | --- | --- | |
12 | | -| `standard` | Fast and affordable. Works for the vast majority of the web: blogs, docs, news, APIs, marketing pages. | **1 credit** per page | Fast | |
13 | | -| `stealth` | Bypasses aggressive bot walls. Use when `standard` gets blocked. | **3 credits** per page | Slower | |
14 | | -| `auto` (default) | Reader picks. Starts with `standard` and escalates to `stealth` only if a block is detected. | Billed per mode actually used | Varies | |
| 12 | +| `standard` (default) | Fast and affordable. Works for the vast majority of the web: blogs, docs, news, APIs, marketing pages. | **1 credit** per page | Fast | |
| 13 | +| `premium` | Bypasses aggressive bot walls. Use when `standard` gets blocked. | **3 credits** per page | Slower | |
15 | 14 |
|
16 | 15 | You set the mode with `proxyMode` on the request: |
17 | 16 |
|
18 | 17 | ```json |
19 | 18 | { |
20 | 19 | "url": "https://shop.example.com/item/42", |
21 | | - "proxyMode": "auto" |
| 20 | + "proxyMode": "premium" |
22 | 21 | } |
23 | 22 | ``` |
24 | 23 |
|
25 | | -Omit `proxyMode` entirely and Reader defaults to `auto`. This is the recommended default for 95% of use cases. |
| 24 | +Omit `proxyMode` entirely and Reader defaults to `standard`. |
26 | 25 |
|
27 | | -## How `auto` works |
| 26 | +## When to use `premium` |
28 | 27 |
|
29 | | -When you request `auto`, Reader tries `standard` first. If the page comes back clean, you're billed 1 credit and done. If the response looks like a block (a CAPTCHA, a 403 with a challenge page, a CDN bot wall), Reader retries with `stealth` automatically and you're billed 3 credits for that page. |
| 28 | +Force `proxyMode: "premium"` when the target site actively blocks datacenter proxies. Amazon product pages, LinkedIn profiles, booking sites, many e-commerce platforms, and most large social networks require premium residential proxies to get clean content. |
30 | 29 |
|
31 | | -You can see what actually happened in the response: |
32 | | - |
33 | | -```json |
34 | | -{ |
35 | | - "data": { |
36 | | - "url": "https://shop.example.com/item/42", |
37 | | - "markdown": "...", |
38 | | - "metadata": { |
39 | | - "proxyMode": "stealth", |
40 | | - "proxyEscalated": true, |
41 | | - "duration": 2341 |
42 | | - } |
43 | | - } |
44 | | -} |
| 30 | +```ts |
| 31 | +await reader.read({ |
| 32 | + url: "https://www.amazon.com/dp/B08N5WRWNW", |
| 33 | + proxyMode: "premium", |
| 34 | +}); |
45 | 35 | ``` |
46 | 36 |
|
47 | | -- `metadata.proxyMode` is **always the mode that actually ran**: `"standard"` or `"stealth"`, never `"auto"`. |
48 | | -- `metadata.proxyEscalated` is `true` only when `auto` escalated from standard to stealth. For explicit mode requests it's always `false`. |
49 | | - |
50 | | -Track `proxyEscalated` in your logs if you want to understand which sites are bot-walled; they'll flag themselves. |
| 37 | +## When to use `standard` |
51 | 38 |
|
52 | | -## When to override `auto` |
| 39 | +`standard` is the default and works for the vast majority of the web. Use it explicitly when you want to guarantee cost at exactly 1 credit per page, or when you want a clear error signal if a site blocks you (rather than silently paying more). |
53 | 40 |
|
54 | | -Most callers never should. But there are two cases where it helps: |
55 | | - |
56 | | -**Force `stealth` when you already know the site is hostile.** Amazon product pages, LinkedIn profiles, booking sites, many e-commerce platforms, most "big social" networks: `auto` will eventually land on `stealth` for these anyway, but forcing it skips the wasted `standard` attempt and gets you clean data on the first try. |
57 | | - |
58 | | -**Force `standard` when you want to guarantee the cheaper tier.** If you're running a 100k-URL batch against a known-friendly site (your own blog, a partner's docs, a public API) and you want to cap spend at exactly 1 credit per page, set `proxyMode: "standard"` explicitly. You'll get an error response instead of a `stealth` escalation if anything does get blocked, and you can decide what to do case by case. |
59 | | - |
60 | | -For a deeper decision guide, see [Choosing a proxy mode](/home/guides/advanced/choosing-a-proxy-mode). |
| 41 | +```ts |
| 42 | +await reader.read({ |
| 43 | + url: "https://docs.example.com/api/v1/reference", |
| 44 | + proxyMode: "standard", |
| 45 | +}); |
| 46 | +``` |
61 | 47 |
|
62 | 48 | ## Pre-flight credit checks |
63 | 49 |
|
64 | 50 | Reader checks your balance before the scrape runs: |
65 | 51 |
|
66 | | -- **Explicit `stealth`**: rejected upfront if your balance is below 3 credits per page. |
67 | | -- **`standard` or `auto`**: rejected upfront if your balance is below 1 credit per page (the optimistic cost). |
68 | | - |
69 | | -If `auto` escalates mid-request and the true cost comes out to 3x what the pre-flight estimated, the charge still goes through, and your balance can briefly go negative on the last request of a burst. Your next request gets a `402 insufficient_credits` error until you top up or your credits reset. This matches how most metered billing systems work. |
| 52 | +- **`premium`**: rejected upfront if your balance is below 3 credits per page. |
| 53 | +- **`standard`**: rejected upfront if your balance is below 1 credit per page. |
70 | 54 |
|
71 | 55 | ## Cache hits are free |
72 | 56 |
|
73 | | -A scrape served from cache is **0 credits regardless of mode**. If the cached version was captured with `standard` and you now ask for it with `stealth`, Reader still serves the cached copy and bills you nothing. The mode you requested only matters when Reader has to actually fetch the page. |
| 57 | +A scrape served from cache is **0 credits regardless of mode**. If the cached version was captured with `standard` and you now ask for it with `premium`, Reader still serves the cached copy and bills you nothing. The mode you requested only matters when Reader has to actually fetch the page. |
74 | 58 |
|
75 | 59 | See [Caching](/home/concepts/caching) for how keys and TTLs work. |
76 | 60 |
|
|
0 commit comments