Skip to content

WEB-ENGINE-UPLIFT.md

Latest commit

 

History

History
224 lines (204 loc) · 14 KB

WEB-ENGINE-UPLIFT.md

File metadata and controls

224 lines (204 loc) · 14 KB
Raw

gather web-data engine uplift

Make gather the web-data engine that offers the SUPERSET of the best features of browser-use, Scrapling, crawlee, and firecrawl, is more performant than each on its own ground, AND is the only one whose every operation carries a re-verifiable receipt. Then wire those capabilities into learn, forum, and index.

Capability is the price of entry, not the differentiator. The receipt is a MULTIPLIER on top of being the most capable and fastest choice; it is never a substitute for a feature a user actually needs. If a competitor does something users pick it for, gather must do that thing at least as well (zero-dep where it can, via an optional capability backend where it must) and additionally witness it. "Honest but less capable" is a losing position and is out of scope.

The competitors and the gap (verified 2026-07-02)

Tool Core strength What it cannot prove
browser-use LLM-driven Playwright automation that its output was not hallucinated
Scrapling adaptive element relocation, TLS-impersonation fetch, ~2ms lxml parse that a relocated element is still the right one
crawlee persistent queue, proxy/session/fingerprint, autoscale that a crawled page is unaltered
firecrawl scrape to markdown, crawl, map, schema extract which extracted field came from the page vs the model

gather's wedge is the receipt: content hashes, selector-to-source-node provenance, an enforced fetched-vs-inferred boundary, and drift verdicts that make silent breakage visible.

The persistent goal

/goal Make gather the accountable web-data engine that out-features and
out-performs browser-use, Scrapling, crawlee, and firecrawl, and wire its
outputs as first-class receipts into learn, forum, and index. Do NOT stop, wind
down, ask permission to continue, or declare done until every wedge below is
shipped as real, tested, zero-dep-core code (an optional browser backend is the
lone allowed edge), each on its own branch and PR-ready, each carrying a
re-verifiable receipt none of the four competitors can produce. After finishing
a wedge, immediately start the next in priority order. Only pause for a genuine
operator-only blocker (a secret, an external paid service, or a real design
fork); surface it in one line and continue on the next unblocked wedge. Advance
the gather-web-uplift progress marker each cycle.

Non-negotiables: zero core deps (stdlib only) with any browser strictly optional
and capability-gated; when a capability is missing, degrade to an honest
UNVERIFIABLE, never fake it; every wedge ships real tests with meaningful
assertions plus a must-fail negative fixture; no secrets, synthetic fixtures
only; no em-dashes in project text; honest scope labels; never merge without
operator authorization.

Wedges (done = best-in-class capability + tests + negative fixture + docs +
version bump + PR-ready; EVERY wedge ALSO emits its receipt):
  1. EXTRACT: HTML to Markdown/text/structured, CSS-lite selection, adaptive
     element relocation. Match firecrawl markdown + Scrapling selectors.
     [receipt: per-block path+hash, MATCH/RELOCATED/DRIFT/GONE]
  2. FETCH: conditional GET/ETag, retry/backoff, redirect provenance, HTTP
     concurrency, on-disk response cache (dev mode). Match Scrapling.Fetcher.
     [receipt: bytes + headers digest, redirect chain]
  3. CRAWL + MAP: concurrent resumable crawler, frontier BFS/DFS, sitemap +
     robots, URL canonicalization/dedup, depth/page limits, per-host throttle.
     Match crawlee queue + firecrawl crawl/map. [receipt: witnessed crawl ledger]
  4. STRUCTURED EXTRACT: schema to JSON via CSS/XPath/regex selectors with an
     adaptive-relocation fallback; every field bound to a source node + hash; a
     REJECT rule forbids any value not present in fetched content. Match firecrawl
     extract, beat it on precision. [receipt: field-to-node binding]
  5. CAPABILITY BACKENDS (the parity layer; all optional + capability-gated, with
     a stdlib fallback or an honest UNVERIFIABLE when absent, never a fake):
       a. BROWSER: JS render, click/fill/scroll, screenshot. Match browser-use +
          Scrapling Dynamic.
       b. STEALTH: TLS/browser impersonation transport, proxy rotation, session +
          fingerprint persistence, Cloudflare handling. Match Scrapling stealth +
          crawlee sessions.
       c. FAST PARSE: optional lxml/selectolax backend to win raw parse speed;
          stdlib stays the default and the fallback.
     [receipt: which backend + capability level produced each artifact]
  6. SEARCH + AGENT INTAKE: web-search-to-content and a URL-less gather agent
     loop. Match firecrawl search + agent. [receipt: query + source set]
  7. DX + PERFORMANCE: interactive CLI shell, dev-mode cache, pause/resume
     everywhere, JSON/JSONL exports, and MCP tools for all of the above; publish
     an honest benchmark table (fast backend on and off) against the competitors'
     numbers. Win on speed with the fast backend; state the zero-dep number too.
  8. INTEROP: gather crawl/corpus to index context-envelope + graph feed; gather
     extraction to a forum evidence lane; gather selector-provenance to a learn
     proof-lesson. Each interop carries the best capability of that flagship's
     category, not only the receipt. Proven end-to-end on the organ-bundle spine.

Organically complete when the capability superset above is shipped and green (a
user can pick gather over any single competitor on features AND speed AND
verifiability), the three interop demos pass end-to-end, docs + version bumped,
and an honest benchmark table is published. Then, and only then, stop.

Status

  • Wedge 1: DONE on branch feat/accountable-extract-track.
    • src/gather/dom.py — zero-dep HTML DOM, stable node paths, CSS-lite select.
    • src/gather/extract.py — HTML to Markdown plus a re-verifiable Extraction receipt (content hash, per-block path + hash, fetched-vs-inferred method).
    • src/gather/track.py — fingerprint + relocate emitting the closed verdict set MATCH / RELOCATED / DRIFT / GONE with a residual.
    • Tests: tests/test_dom.py, tests/test_extract.py, tests/test_track.py (16 new; full suite 305 passed), including tamper and drift negatives.
  • Wedge 2: DONE on the same branch.
    • src/gather/fetch.py — an accountable HTTP GET that reuses net.py's SSRF guard and cross-origin credential stripping, and returns a FetchReceipt (bytes hash, headers digest, recorded redirect chain, status, attempts) with conditional GET (ETag / If-Modified-Since, honest 304) and retry/backoff. The transport is a seam, so retry/conditional/receipt logic is tested offline.
    • Tests: tests/test_fetch.py (7; full suite 312 passed), including retry, exhaustion, tamper, and routing-header-guard negatives.
    • Honest limitation vs Scrapling: default UA identifies gather (no browser impersonation) and zero-dep cannot forge a TLS fingerprint; a caller may supply their own headers, on the record.
  • Wedge 3: DONE on the same branch.
    • src/gather/crawl.py — a competitive crawler (concurrent wave fetching, BFS/DFS frontier, URL canonicalization + dedup, robots.txt via stdlib robotparser, sitemap discovery, depth/page caps, per-host throttle, and pause/resume via a serializable CrawlState) PLUS an append-only, hash-chained CrawlLedger a reviewer can re-derive to prove the crawl was not reordered, truncated, or edited.
    • Tests: tests/test_crawl.py (11; full suite 323 passed), including dedup, depth/page caps, robots block, sitemap seeding, resume-across-pause chain continuity, concurrent-workers parity, and ledger tamper detection.
  • Wedge 4: DONE on the same branch.
    • src/gather/schema_extract.py — extract_schema (schema of CSS-lite selectors + optional attr/regex/many to a JSON record, each field bound to a source node path + hash; the firecrawl extract shape) AND verify_record, the hallucination-REJECT: any value in a proposed record not grounded in the fetched content is rejected. Turns LLM extraction from trust-the-model into prove-it-against-the-source.
    • Tests: tests/test_schema_extract.py (6; full suite 329 passed), including tamper detection and a hallucinated-field rejection negative.
  • Wedge 5-core: DONE on the same branch (the accountable half).
    • src/gather/backends.py — a capability registry + gating: backends declare capabilities (js-render, stealth, fast-parse); render() resolves the best available and, when a required capability has no backend, returns UNVERIFIABLE with a reason and never a faked render. Every result records which backend served it. best_parser/detect_fast_parse select a native parser when installed, else stdlib.
    • Tests: tests/test_backends.py (7; full suite 336 passed), including the honest-degrade negative (missing capability -> UNVERIFIABLE, not a fake).
  • Wedge 5-backends: DONE on the same branch (opt-in extras; core stays zero-dep).
    • src/gather/fastparse.py — lxml fast-parse producing the IDENTICAL gather Node tree (paths match stdlib); verified ~2x faster (33 ms vs 69 ms on the ~15k-element bench).
    • src/gather/backends_browser.py — Playwright js-render backend. Verified: it really launches headless Chromium and executes JavaScript here. Missing browser binary degrades to honest UNVERIFIABLE, never a fake.
    • src/gather/backends_stealth.py — curl_cffi TLS-impersonation transport for the accountable fetch path (same FetchReceipt), with the SSRF guard re-applied per redirect hop and cross-origin credential stripping.
    • pyproject.toml extras: fast / browser / stealth / all.
    • Tests: tests/test_fastparse.py, tests/test_backends_browser.py, tests/test_backends_stealth.py, plus the render-failure-degrade negative (18 across the group; full suite 356 passed).
  • Wedge 6: DONE (src/gather/search.py). Pluggable search-provider seam -> SearchReceipt of SOURCE_LEADs; honest UNVERIFIABLE with no provider; search_and_fetch chains leads into the accountable fetch path; searx_provider is a no-API-key SearXNG backend. firecrawl search/agent parity.
  • Wedge 7 (DX + performance): DONE, on the same branch.
    • src/gather/cache.py — dev-mode response cache (Scrapling dev-mode parity): content-addressed store, offline replay, and conditional revalidation that serves the cached body on a 304.
    • src/gather/export.py — uniform JSON / JSONL export across every receipt.
    • examples/bench.py — an honest micro-benchmark, plus a real perf fix: node paths are now assigned at parse time, so extract over ~15k elements went from 1171 ms to 113 ms (about 10x). See Benchmarks below.
    • Tests: tests/test_cache.py + tests/test_export.py (9; full suite 345 passed), including cache tamper detection and dev-mode replay-without-refetch.
    • CLI wiring: gather caps (honest capability report), gather extract, gather markdown, gather crawl (src/gather/web_commands.py), each printing a receipt as JSON.
  • Wedge 8 (interop): DONE (src/gather/interop.py). Maps gather's receipts into organ-bundle interchange entries; verified to validate against proof-surface's real organ_receipt_bundle validator (0 issues), so gather composes on the same spine as index/forum/learn/crucible/emet. crawl_entry reflects ledger integrity (tampered chain -> failed entry).

Benchmarks (honest, zero-dep)

Measured by examples/bench.py over a ~461 KB, ~5,000-element document. Each row is an INTERVAL (median, then min/max) over 9 iterations, not a single boast; a single number hides its own variance. gather.benchmark.run_suite() returns the whole evidence artifact (schema, environment, document, per-op stats), and python examples/bench.py --out bench.json writes it, so a reader can reproduce and compare like with like. Absolute milliseconds are machine-specific; the interval and the recorded environment are what make the numbers honest.

op median min max
parse_dom (stdlib, zero-dep) ~69 ms ~62 ms ~80 ms
select('.r') (5000 hits) ~14 ms ~13 ms ~14 ms
to_markdown ~93 ms ~78 ms ~95 ms
extract (markdown + per-block receipt) ~120 ms ~108 ms ~125 ms
parse_dom (lxml fast-parse backend) ~40 ms ~28 ms ~42 ms

Reproduce: python examples/bench.py --iters 9 (add --out bench.json for the evidence artifact). Numbers above are from one machine; re-run yours to compare.

Honest comparison: Scrapling publishes ~2 ms text extraction on 5,000 nested elements using lxml (a C parser). gather's stdlib parser is real and usable but is not going to match a C parser on raw speed. That is precisely what the optional fast-parse backend (wedge 5-backends, lxml/selectolax, gated on the dependency decision) is for; the stdlib path stays the zero-dep default and fallback. gather already wins on what it carries that lxml does not: a re-verifiable per-block provenance receipt.

Streaming (the partialupdate integration)

src/gather/stream.py folds in the idea from Phil Holden's partialupdate (the talk "What if AI replies in HTML not Markdown?"): parse an HTML stream incrementally and commit only the STABLE part, holding back the incomplete tail so a progressive render never flashes half-open markup. The repo is gone (deleted from GitHub, never on npm, no Wayback snapshot), but the concept is clear and gather is its natural home.

gather adds the accountability layer the streaming libraries lack: as each block element closes it is committed with a content hash and linked into an append-only hash chain (StreamLedger), so a streamed extraction is a re-verifiable ledger of stable increments, and the still-open tail is explicitly PENDING (not yet verifiable), never treated as final. Chunk boundaries do not matter: feeding a document in any split yields the same commits, hashes, and order as feeding it whole. This is extract.py's stable-vs-incomplete honesty applied to a live stream, and it directly serves the case the talk is about: an AI streaming HTML, extracted and receipted as it arrives.