Skip to content

web-search.md

Latest commit

 

History

History
95 lines (66 loc) · 8.24 KB

web-search.md

File metadata and controls

95 lines (66 loc) · 8.24 KB
Raw

Runtime web search and fetch

Runtime agents can discover public references with web_search and read them with web_fetch (ADR-0079, extended by ADR-0080). Both tools are optional and agent-directed. No search, fetched page or memory content is loaded automatically.

Project settings

In Administration → Projects → project → Web search & fetch:

  1. Enable outbound search and select the permitted runtime agents.
  2. Optionally enable Allow agents to read public web pages.
  3. Save. New grants apply when an agent next executes.

Configuration requires project.web-search.manage. Projects start disabled with designer preselected. Fetch is separately disabled by default, including on projects that previously enabled search. Agents with an empty tool list receive neither tool. New agents are never added automatically.

GET /api/admin/projects/:projectId/web-search returns { enabled, fetchEnabled, agents } inside the normal response envelope. PUT accepts these fields; omitted fetchEnabled defaults to false for older clients. Migration 040_web_fetch.sql extends migration 039. No provider key, environment setting or additional service is needed. Changes are audited without query content.

The worker rechecks live permission on every call and before returning its result. Saving any settings change invalidates in-flight responses, even if the capability is re-enabled before a request finishes. Data already returned remains in the conversation. These permissions do not affect existing shell or MCP network capabilities.

Search

{
  "query": "PostgreSQL transaction isolation",
  "maxResults": 10,
  "site": "postgresql.org",
  "timeRange": "y",
  "region": "us-en"
}
Argument Meaning
query Required, 1–1,000 trimmed characters, without control or formatting characters
maxResults Optional, 1–10; defaults to 10
site Optional public domain without a scheme, path or wildcard; results must belong to it or its subdomains
timeRange Optional DuckDuckGo date hint: d, w, m or y
region Optional DuckDuckGo region code, such as us-en, vn-vi or wt-wt

Results contain provider, the original query, retrievedAt, results (title, URL, snippet), truncated, an untrusted-data notice, and a budget with remaining step/run reservations. Quota values are point-in-time counts: parallel children may reserve more afterward. Marked advertisements, duplicate URLs and recognizable unsafe links are omitted. Results may be fewer than requested. Site restrictions are enforced on returned hostnames; date and region are provider hints, not certified metadata or publication dates.

Snippets are not full-page evidence. Use web_fetch for details or fetch a known reference URL directly if search is unavailable and fetch is permitted. There is no automatic provider fallback, pagination request, CAPTCHA solving or retry loop.

Fetch

{"url":"https://www.postgresql.org/docs/current/transaction-iso.html","maxChars":16000}
Argument Meaning
url Required public HTTP/HTTPS URL, at most 4,096 characters; no credentials, common secret/signed query parameters or nonstandard port
maxChars Optional, 1,000–32,000; defaults to 16,000, additionally bounded by response bytes
offset Optional character offset, defaults to zero
expectedHash Optional SHA-256 returned by the previous fetch; required when offset is greater than zero

Fetch returns:

  • url, finalUrl, title, contentType, and UTC retrievedAt (not publication time).
  • content: readable extracted text, with code blocks, headings, lists and tables where available. HTML scripts/forms/navigation and hidden elements are removed; nothing executes.
  • contentHash: SHA-256 of the bounded extracted text, not the original HTML.
  • offset, nextOffset, totalChars, truncated, and sourceTruncated.
  • Up to 20 reference links, an untrusted-data notice, and remaining step/run budget.

If nextOffset is non-null, call again with the same URL, that offset, and expectedHash equal to contentHash. Every continuation downloads/extracts again and consumes a fetch reservation. A different hash returns a changed-content error; restart at offset zero. sourceTruncated means the 400,000-character extraction cap was reached: the rest is not retained, so use a more focused source. There is no stored page content or cross-call cache. Links are returned as data and are validated again if fetched; DNS is not resolved just to list a link.

HTML/XHTML, plain text, Markdown and JSON are supported. PDF, images, downloads, OCR and JavaScript rendering are not. No cookies, caller headers, authentication or browser session are available. A public login page or application shell may yield text without the desired document.

Budgets and concurrency

Limit Search Fetch
Reservations per logical step 20 40
Reservations per top-level run, shared by children 100 200
Active requests per project 1 3
Minimum interval between starts 2 seconds No extra delay
Deadline including DNS/redirects/body 25 seconds 30 seconds
Redirects None At most 4; each hop checked
Wire and decompressed body 1 MB each 2 MB each
Tool response 64 KB 64 KB

Quotas are independent between search and fetch. The worker atomically reserves in Postgres before network access. Failed/cancelled attempts count; argument, policy, concurrency and exhausted-budget denials before reservation do not. Redirects within one fetch count as one logical reservation, not as additional fetch calls. Leases expire after 45 seconds if a worker dies. Reservations survive retries and restarts; logical step keys survive newly allocated child run IDs. Re-executed steps do not reset quota.

Agent summaries count search and fetch tool calls separately, including unsuccessful service calls; these differ from upstream reservations. Live external inputs cause participating steps/workflows to execute again on retry rather than reuse stale results; the retry's answer names the agents with web lookups, the portal says their steps run again, and the run's log says so for each such step (ADR-0090).

Errors

Errors distinguish disabled capability, exhausted quota, project concurrency/cooldown, unsafe URL, blocked DNS destination, redirects, unsupported content, size limits, changed content, rate limiting and unavailable sources. DuckDuckGo can challenge or change its HTML format. Explicit no-results pages return an empty result list; an unrecognized page shape returns an unavailable error. Agents should do other work or use another public reference rather than repeatedly retrying a blocked source.

Network and data handling

The worker resolves all destination addresses and rejects private, loopback, link-local, metadata, multicast and reserved networks, including mapped/transition IPv6. Connections pin a validated address while retaining the original Host and TLS certificate checks. Every redirect gets URL/DNS validation; HTTPS downgrades are blocked. Requests use no ambient cookies, credentials, proxy settings or caller headers. Encoded and decoded bodies, headers, redirects and elapsed time are bounded; cancellation destroys the active connection.

Queries go to DuckDuckGo; fetch URLs and the worker's outbound IP are visible to destination sites. Use public technical terms and public references. Credential-pattern checks cannot recognize every confidential phrase, opaque token or signed path. Existing tool events, transcripts and telemetry may record redacted arguments and results under normal retention. Reservation rows store only identifiers, operation, timestamps and lease/completion state.

External pages may contain false claims or prompt injection. Ignore their instructions, cite source URLs and distinguish external guidance from project facts. Project documents and pinned repository evidence remain authoritative. Neither tool writes memory automatically or certifies repository evidence. LLM tokens for queries, results and page text remain billable even though these tools need no provider key.