Skip to content

Commit 10e16b2

Browse files
nihalwasherenihalwashere
committed
Add CLI docs, screenshot format, update Hero references to Playwright
1 parent a561749 commit 10e16b2

18 files changed

Lines changed: 295 additions & 43 deletions

‎home/cli.mdx‎

Lines changed: 203 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,203 @@
1+
---
2+
title: "CLI"
3+
description: "Read the web from your terminal. Built for AI coding agents."
4+
---
5+
6+
The Reader CLI lets you scrape, crawl, and take screenshots from the command line. It's a thin wrapper around the [Reader API](/api-reference/read) - no local browser needed.
7+
8+
Built for AI coding agents like Claude Code, Cursor, and Codex, but works great for humans too.
9+
10+
## Install
11+
12+
```bash
13+
npm install -g @vakra-dev/reader-cli
14+
```
15+
16+
Or run directly without installing:
17+
18+
```bash
19+
npx @vakra-dev/reader-cli scrape https://example.com
20+
```
21+
22+
## Authentication
23+
24+
<Tabs>
25+
<Tab title="For humans">
26+
Run once to save your API key:
27+
```bash
28+
reader config set api-key rdr_your_key_here
29+
```
30+
The key is saved to `~/.reader/config.json` and used for all future commands.
31+
</Tab>
32+
<Tab title="For AI agents / CI">
33+
Set the environment variable:
34+
```bash
35+
export READER_API_KEY=rdr_your_key_here
36+
```
37+
The env var takes precedence over the config file. No disk writes needed.
38+
</Tab>
39+
</Tabs>
40+
41+
Get your API key from the [Reader dashboard](https://console.reader.dev).
42+
43+
## Scrape a page
44+
45+
```bash
46+
reader scrape https://example.com
47+
```
48+
49+
Output is clean markdown, printed to stdout. Pipe it anywhere:
50+
51+
```bash
52+
reader scrape https://docs.stripe.com/payments > stripe-payments.md
53+
```
54+
55+
### Formats
56+
57+
```bash
58+
reader scrape https://example.com # markdown (default)
59+
reader scrape https://example.com -f html # cleaned HTML
60+
reader scrape https://example.com -f screenshot -o page.png # full-page PNG
61+
```
62+
63+
### JSON output
64+
65+
Get the full API response with metadata:
66+
67+
```bash
68+
reader scrape https://example.com --json
69+
```
70+
71+
```json
72+
{
73+
"url": "https://example.com",
74+
"markdown": "# Example Domain\n\n...",
75+
"metadata": {
76+
"duration": 892,
77+
"cached": false,
78+
"proxyMode": "standard"
79+
},
80+
"pageMetadata": {
81+
"title": "Example Domain"
82+
}
83+
}
84+
```
85+
86+
### Options
87+
88+
```bash
89+
reader scrape <url> [options]
90+
91+
Options:
92+
-f, --format <format> markdown (default), html, screenshot
93+
--json Full JSON response
94+
-o, --output <file> Write to file
95+
--no-main-content Include nav, header, footer
96+
--include-tags <sel> CSS selectors to keep (comma-separated)
97+
--exclude-tags <sel> CSS selectors to remove (comma-separated)
98+
--wait-for <selector> Wait for element before capturing
99+
--timeout <ms> Timeout in milliseconds (default: 30000)
100+
--proxy-mode <mode> standard, stealth, auto
101+
```
102+
103+
## Crawl a site
104+
105+
Discover and scrape all pages on a website:
106+
107+
```bash
108+
reader crawl https://docs.example.com
109+
```
110+
111+
Each page's markdown is output separated by `---`. Save to individual files:
112+
113+
```bash
114+
reader crawl https://docs.example.com -o ./docs/
115+
```
116+
117+
### URL discovery only
118+
119+
List all discovered URLs without scraping content:
120+
121+
```bash
122+
reader crawl https://docs.example.com --urls-only
123+
```
124+
125+
### Options
126+
127+
```bash
128+
reader crawl <url> [options]
129+
130+
Options:
131+
--max-depth <n> Crawl depth (default: 2)
132+
--max-pages <n> Max pages (default: 20)
133+
--urls-only Only list URLs, don't scrape
134+
--json Full JSON response
135+
-o, --output-dir <dir> Write each page to a separate file
136+
```
137+
138+
## Check status
139+
140+
Verify your setup and see your credit balance:
141+
142+
```bash
143+
reader status
144+
```
145+
146+
```
147+
Reader CLI v0.1.0
148+
API: https://api.reader.dev
149+
Key: rdr_...c3bb
150+
Credits: 874 / 1000 (free tier)
151+
Resets: 2026-07-01
152+
```
153+
154+
## Check credits
155+
156+
```bash
157+
reader credits
158+
```
159+
160+
```
161+
Balance: 874 / 1000
162+
Used: 126
163+
Tier: free
164+
Resets: 2026-07-01T00:00:00.000Z
165+
```
166+
167+
## Configuration
168+
169+
```bash
170+
reader config set api-key <key> # save API key
171+
reader config set api-url <url> # custom API URL
172+
reader config show # show current config
173+
```
174+
175+
Config is stored at `~/.reader/config.json`. Environment variables always take precedence:
176+
177+
| Setting | Env var | Config key |
178+
| --- | --- | --- |
179+
| API key | `READER_API_KEY` | `apiKey` |
180+
| API URL | `READER_API_URL` | `apiUrl` |
181+
182+
## For AI agents
183+
184+
The CLI is designed to work seamlessly with AI coding agents:
185+
186+
- **Markdown to stdout** - agents read the content directly
187+
- **Errors to stderr** - never pollutes piped output
188+
- **Exit codes** - 0 on success, 1 on error
189+
- **No interactive prompts** - everything via flags and env vars
190+
- **`--json` flag** - structured output for programmatic use
191+
192+
Example in a Claude Code or Cursor workflow:
193+
194+
```bash
195+
# Agent reads documentation to answer a question
196+
reader scrape https://docs.stripe.com/payments/quickstart
197+
198+
# Agent crawls a site for context
199+
reader crawl https://docs.example.com --max-pages 10 -o ./context/
200+
201+
# Agent takes a screenshot for visual analysis
202+
reader scrape https://example.com -f screenshot -o page.png
203+
```

‎home/concepts/formats-and-extraction.mdx‎

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -13,15 +13,16 @@ Pass `formats` on the request to choose what comes back:
1313
| --- | --- |
1414
| `"markdown"` (default) | Clean, structured markdown. Headings, lists, links, code blocks preserved. Best for LLMs and RAG. |
1515
| `"html"` | The cleaned HTML Reader used as the source for markdown conversion. Useful when you need to run your own DOM parsing. |
16+
| `"screenshot"` | A full-page screenshot as a base64-encoded PNG. Useful when AI agents need visual context. |
1617

1718
Every response also includes `rawHtml` - the unprocessed HTML exactly as the browser rendered it, before any cleaning or content extraction. This is always returned regardless of `formats`.
1819

19-
You can request both:
20+
You can request any combination:
2021

2122
```json
2223
{
2324
"url": "https://example.com",
24-
"formats": ["markdown", "html"]
25+
"formats": ["markdown", "screenshot"]
2526
}
2627
```
2728

@@ -33,12 +34,16 @@ The response includes whichever fields you asked for:
3334
"url": "https://example.com",
3435
"rawHtml": "<html><head>...</head><body>...</body></html>",
3536
"markdown": "# Example Domain\n\n...",
36-
"html": "<h1>Example Domain</h1>...",
37+
"screenshot": "iVBORw0KGgoAAAANSUhEUgAA...",
3738
"metadata": { /* ... */ }
3839
}
3940
}
4041
```
4142

43+
<Note>
44+
Screenshots are full-page captures. They bypass the cache since each capture is unique. The `screenshot` field is a base64-encoded PNG string that you decode on the client side.
45+
</Note>
46+
4247
## Main content extraction
4348

4449
By default Reader strips away navigation, footers, sidebars, cookie banners, newsletter pop-ups, and other boilerplate, keeping just the article body. This is the `onlyMainContent: true` default.

‎home/concepts/read-primitive.mdx‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -88,7 +88,7 @@ Use the `id` to poll `GET /v1/jobs/{id}`, stream progress with SSE, or subscribe
8888

8989
You tell Reader **what** to fetch. Reader decides **how**:
9090

91-
- How to render the page (full browser with JavaScript execution and TLS fingerprinting).
91+
- How to render the page (full browser with JavaScript execution and stealth evasion).
9292
- Whether to escalate the proxy from datacenter to residential when a block is detected (see [Proxy modes](/home/concepts/proxy-modes)).
9393
- Whether to serve from cache.
9494
- How to parallelize a batch.

‎mint.json‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -65,7 +65,8 @@
6565
"group": "Get Started",
6666
"pages": [
6767
"home/introduction",
68-
"home/quickstart"
68+
"home/quickstart",
69+
"home/cli"
6970
]
7071
},
7172
{

‎openapi.json‎

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -141,6 +141,10 @@
141141
"html": {
142142
"type": "string"
143143
},
144+
"screenshot": {
145+
"type": "string",
146+
"description": "Base64-encoded PNG screenshot of the full page. Only present when \"screenshot\" is included in formats."
147+
},
144148
"metadata": {
145149
"$ref": "#/components/schemas/ScrapeMetadata"
146150
}
@@ -302,7 +306,8 @@
302306
"type": "string",
303307
"enum": [
304308
"markdown",
305-
"html"
309+
"html",
310+
"screenshot"
306311
]
307312
},
308313
"description": "Content formats to include in the response.",
@@ -506,6 +511,10 @@
506511
"html": {
507512
"type": "string"
508513
},
514+
"screenshot": {
515+
"type": "string",
516+
"description": "Base64-encoded PNG screenshot."
517+
},
509518
"statusCode": {
510519
"type": "integer"
511520
},

‎sdk/javascript.mdx‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -67,6 +67,23 @@ if (result.kind === "scrape") {
6767
}
6868
```
6969

70+
### Screenshots
71+
72+
Request a screenshot alongside content:
73+
74+
```typescript
75+
const result = await reader.read({
76+
url: "https://example.com",
77+
formats: ["markdown", "screenshot"],
78+
});
79+
80+
if (result.kind === "scrape" && result.data.screenshot) {
81+
// screenshot is a base64-encoded PNG
82+
const buffer = Buffer.from(result.data.screenshot, "base64");
83+
fs.writeFileSync("screenshot.png", buffer);
84+
}
85+
```
86+
7087
### Multiple URLs (batch)
7188

7289
Passing `urls` creates an async job. The SDK auto-polls until the job terminates and returns `{ kind: "job", data: Job }` with all results collected across pagination.

‎sdk/python.mdx‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -78,6 +78,25 @@ if result.kind == "scrape":
7878
print(result.data.metadata.proxy_escalated) # True only if auto escalated
7979
```
8080

81+
### Screenshots
82+
83+
Request a screenshot alongside content:
84+
85+
```python
86+
import base64
87+
88+
result = reader.read(
89+
url="https://example.com",
90+
formats=["markdown", "screenshot"],
91+
)
92+
93+
if result.kind == "scrape" and result.data.screenshot:
94+
# screenshot is a base64-encoded PNG
95+
png_bytes = base64.b64decode(result.data.screenshot)
96+
with open("screenshot.png", "wb") as f:
97+
f.write(png_bytes)
98+
```
99+
81100
### Multiple URLs (batch)
82101

83102
Passing `urls` creates an async job. The SDK auto-polls until the job terminates and returns `ReadResult(kind="job", data=Job)` with all results collected across pagination.

‎self-hosted/api-reference/reader-client.mdx‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ title: "ReaderClient"
33
description: "Constructor, options, methods, and lifecycle for the main Reader API."
44
---
55

6-
`ReaderClient` is the high-level API you'll use for 99% of self-hosted Reader workloads. It owns the HeroCore instance and the browser pool, exposes `scrape()` and `crawl()`, and handles lazy initialization.
6+
`ReaderClient` is the high-level API you'll use for 99% of self-hosted Reader workloads. It owns the Playwright pool and the browser instances, exposes `scrape()` and `crawl()`, and handles lazy initialization.
77

88
## Constructor
99

@@ -45,7 +45,7 @@ See [BrowserPoolConfig](/self-hosted/concepts/browser-pool) and [ProxyConfig](/s
4545
async start(): Promise<void>
4646
```
4747

48-
Pre-warm the client. Initializes HeroCore and the browser pool without running a scrape. Optional - `scrape()` and `crawl()` will initialize automatically if you haven't called `start()`.
48+
Pre-warm the client. Initializes the Playwright pool and browser instances without running a scrape. Optional - `scrape()` and `crawl()` will initialize automatically if you haven't called `start()`.
4949

5050
```typescript
5151
async scrape(options: ScrapeOptions): Promise<ScrapeResult>
@@ -83,7 +83,7 @@ Helpers for checking proxy pool availability. Useful when you want to gate behav
8383
ReaderClient is lazy by design:
8484

8585
1. `new ReaderClient()` - constructor does nothing expensive
86-
2. First call to `scrape()` or `crawl()` - triggers HeroCore startup and browser pool initialization (1-2 seconds)
86+
2. First call to `scrape()` or `crawl()` - triggers Playwright pool startup and browser initialization (1-2 seconds)
8787
3. Subsequent calls - reuse the warm pool
8888
4. Auto cleanup on `SIGTERM`/`SIGINT`/process exit
8989
5. Explicit `close()` - tears down browsers immediately

‎self-hosted/api-reference/scrape-result.mdx‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,7 @@ interface WebsiteScrapeResult {
2525
metadata: {
2626
baseUrl: string;
2727
statusCode: number;
28-
engine: "hero";
28+
engine: "playwright";
2929
totalPages: number;
3030
scrapedAt: string; // ISO timestamp
3131
duration: number; // milliseconds
@@ -42,7 +42,7 @@ interface WebsiteScrapeResult {
4242
| `html` | Cleaned HTML output (if `"html"` in formats) |
4343
| `metadata.baseUrl` | The original URL that was scraped |
4444
| `metadata.statusCode` | HTTP status returned by the server |
45-
| `metadata.engine` | Engine used (`"hero"`) |
45+
| `metadata.engine` | Engine used (`"playwright"`) |
4646
| `metadata.duration` | Total time in milliseconds |
4747
| `metadata.scrapedAt` | ISO timestamp when the scrape completed |
4848
| `metadata.website` | Parsed page metadata (title, OG tags, etc.) |

0 commit comments

Comments
 (0)