Skip to content

Commit 5c53e24

Browse files
YahiaBakourYahiaBakour
authored
Merge pull request #72 from context-dot-dev/docs/context-dev-readme-opening
README: lead with what the SDK does
2 parents 727751b + 78920ab commit 5c53e24

1 file changed

Lines changed: 116 additions & 41 deletions

File tree

‎README.md‎

Lines changed: 116 additions & 41 deletions
Original file line numberDiff line numberDiff line change
@@ -3,11 +3,7 @@
33
<!-- prettier-ignore -->
44
[![PyPI version](https://img.shields.io/pypi/v/context.dev.svg?label=pypi%20(stable))](https://pypi.org/project/context.dev/)
55

6-
The Context Dev Python library provides convenient access to the Context Dev REST API from any Python 3.9+
7-
application. The library includes type definitions for all request params and response fields,
8-
and offers both synchronous and asynchronous clients powered by [httpx](https://github.com/encode/httpx).
9-
10-
It is generated with [Stainless](https://www.stainless.com/).
6+
Context.dev is a web scraping API for AI agents and LLMs. This SDK turns any URL into clean, LLM-ready markdown, crawls whole sites, searches the web, takes screenshots and extracts structured JSON against a schema you define, all with one API key. Proxies, JavaScript rendering and anti-bot handling run on Context.dev's side, so there is no headless browser to host.
117

128
## Documentation
139

@@ -22,23 +18,101 @@ pip install context.dev
2218

2319
## Usage
2420

21+
Set `CONTEXT_DEV_API_KEY` to your API key; the client reads it automatically.
22+
23+
### Scrape markdown and HTML
24+
2525
The full API of this library can be found in [api.md](api.md).
2626

2727
```python
28-
import os
2928
from context.dev import ContextDev
3029

31-
client = ContextDev(
32-
api_key=os.environ.get("CONTEXT_DEV_API_KEY"), # This is the default and can be omitted
30+
client = ContextDev()
31+
32+
page = client.web.scrape(
33+
url="https://example.com",
34+
formats={"markdown": True, "html": True},
35+
)
36+
print(page.markdown.data)
37+
print(page.html.data)
38+
```
39+
40+
### Extract structured JSON
41+
42+
```python
43+
from context.dev import ContextDev
44+
45+
client = ContextDev()
46+
47+
page = client.web.scrape(
48+
url="https://example.com",
49+
formats={"json": True},
50+
json_params={
51+
"schema": {
52+
"type": "object",
53+
"properties": {
54+
"title": {"type": ["string", "null"]},
55+
"description": {"type": ["string", "null"]},
56+
},
57+
"required": ["title", "description"],
58+
"additionalProperties": False,
59+
},
60+
},
61+
)
62+
print(page.json_.data)
63+
```
64+
65+
### Extract relevant highlights
66+
67+
Return the passages that answer a question about the page.
68+
69+
```python
70+
from context.dev import ContextDev
71+
72+
client = ContextDev()
73+
74+
page = client.web.scrape(
75+
url="https://example.com",
76+
formats={"highlights": True},
77+
highlights_params={"query": "What is this domain used for?"},
3378
)
79+
print(page.highlights.data)
80+
```
81+
82+
### Take a screenshot
83+
84+
The screenshot is returned as a base64 image data URL.
3485

35-
brand = client.brand.retrieve(
36-
domain="REPLACE_ME",
37-
type="by_domain",
86+
```python
87+
from context.dev import ContextDev
88+
89+
client = ContextDev()
90+
91+
page = client.web.scrape(
92+
url="https://example.com",
93+
formats={"screenshot": True},
3894
)
39-
print(brand.request_id)
95+
print(page.screenshot.data)
4096
```
4197

98+
## What you can do
99+
100+
| Task | Method |
101+
| --- | --- |
102+
| Scrape a URL to markdown, HTML, JSON, highlights or a screenshot | `client.web.scrape` |
103+
| Crawl a site and get every page as markdown | `client.web.web_crawl_md` |
104+
| Map every URL on a domain | `client.web.map_urls` |
105+
| Search the web | `client.web.search` |
106+
| Take a screenshot of a page | `client.web.screenshot` |
107+
| Parse PDFs and documents | `client.parse.handle` |
108+
| Run thousands of URLs as a batch | `client.batch.submit` |
109+
| Watch a page for changes | `client.monitors.create` |
110+
| Look up a company's logo, colors and brand data | `client.brand.retrieve` |
111+
112+
## Use it from an AI agent
113+
114+
Context.dev also ships as a plugin for [Claude](https://github.com/context-dot-dev/claude-plugin), [Cursor](https://github.com/context-dot-dev/cursor-plugin) and [Gemini CLI](https://github.com/context-dot-dev/gemini-cli-context), and as tools for [LangChain](https://github.com/context-dot-dev/langchain-context) and [Haystack](https://github.com/context-dot-dev/context-haystack).
115+
42116
While you can provide an `api_key` keyword argument,
43117
we recommend using [python-dotenv](https://pypi.org/project/python-dotenv/)
44118
to add `CONTEXT_DEV_API_KEY="My API Key"` to your `.env` file
@@ -59,11 +133,11 @@ client = AsyncContextDev(
59133

60134

61135
async def main() -> None:
62-
brand = await client.brand.retrieve(
63-
domain="REPLACE_ME",
64-
type="by_domain",
136+
page = await client.web.scrape(
137+
url="https://example.com",
138+
formats={"markdown": True},
65139
)
66-
print(brand.request_id)
140+
print(page.markdown.data)
67141

68142

69143
asyncio.run(main())
@@ -96,11 +170,11 @@ async def main() -> None:
96170
api_key=os.environ.get("CONTEXT_DEV_API_KEY"), # This is the default and can be omitted
97171
http_client=DefaultAioHttpClient(),
98172
) as client:
99-
brand = await client.brand.retrieve(
100-
domain="REPLACE_ME",
101-
type="by_domain",
173+
page = await client.web.scrape(
174+
url="https://example.com",
175+
formats={"markdown": True},
102176
)
103-
print(brand.request_id)
177+
print(page.markdown.data)
104178

105179

106180
asyncio.run(main())
@@ -124,12 +198,12 @@ from context.dev import ContextDev
124198

125199
client = ContextDev()
126200

127-
brand = client.brand.retrieve(
128-
domain="xxx",
129-
type="by_domain",
201+
page = client.web.scrape(
202+
url="https://example.com",
203+
formats={"markdown": True},
130204
timeout_opts={"milliseconds": 1000},
131205
)
132-
print(brand.timeout_opts)
206+
print(page.markdown.data)
133207
```
134208

135209
## Handling errors
@@ -148,9 +222,9 @@ from context.dev import ContextDev
148222
client = ContextDev()
149223

150224
try:
151-
client.brand.retrieve(
152-
domain="REPLACE_ME",
153-
type="by_domain",
225+
client.web.scrape(
226+
url="https://example.com",
227+
formats={"markdown": True},
154228
)
155229
except context.dev.APIConnectionError as e:
156230
print("The server could not be reached")
@@ -194,9 +268,9 @@ client = ContextDev(
194268
)
195269

196270
# Or, configure per-request:
197-
client.with_options(max_retries=5).brand.retrieve(
198-
domain="REPLACE_ME",
199-
type="by_domain",
271+
client.with_options(max_retries=5).web.scrape(
272+
url="https://example.com",
273+
formats={"markdown": True},
200274
)
201275
```
202276

@@ -206,6 +280,7 @@ By default requests time out after 1 minute. You can configure this with a `time
206280
which accepts a float or an [`httpx.Timeout`](https://www.python-httpx.org/advanced/timeouts/#fine-tuning-the-configuration) object:
207281

208282
```python
283+
import httpx
209284
from context.dev import ContextDev
210285

211286
# Configure the default for all requests:
@@ -220,9 +295,9 @@ client = ContextDev(
220295
)
221296

222297
# Override per-request:
223-
client.with_options(timeout=5.0).brand.retrieve(
224-
domain="REPLACE_ME",
225-
type="by_domain",
298+
client.with_options(timeout=5.0).web.scrape(
299+
url="https://example.com",
300+
formats={"markdown": True},
226301
)
227302
```
228303

@@ -264,14 +339,14 @@ The "raw" Response object can be accessed by prefixing `.with_raw_response.` to
264339
from context.dev import ContextDev
265340

266341
client = ContextDev()
267-
response = client.brand.with_raw_response.retrieve(
268-
domain="REPLACE_ME",
269-
type="by_domain",
342+
response = client.web.with_raw_response.scrape(
343+
url="https://example.com",
344+
formats={"markdown": True},
270345
)
271346
print(response.headers.get('X-My-Header'))
272347

273-
brand = response.parse() # get the object that `brand.retrieve()` would have returned
274-
print(brand.request_id)
348+
page = response.parse()
349+
print(page.markdown.data)
275350
```
276351

277352
These methods return an [`APIResponse`](https://github.com/context-dot-dev/context-python-sdk/tree/main/src/context/dev/_response.py) object.
@@ -285,9 +360,9 @@ The above interface eagerly reads the full response body when you make the reque
285360
To stream the response body, use `.with_streaming_response` instead, which requires a context manager and only reads the response body once you call `.read()`, `.text()`, `.json()`, `.iter_bytes()`, `.iter_text()`, `.iter_lines()` or `.parse()`. In the async client, these are async methods.
286361

287362
```python
288-
with client.brand.with_streaming_response.retrieve(
289-
domain="REPLACE_ME",
290-
type="by_domain",
363+
with client.web.with_streaming_response.scrape(
364+
url="https://example.com",
365+
formats={"markdown": True},
291366
) as response:
292367
print(response.headers.get("X-My-Header"))
293368

0 commit comments

Comments
 (0)