pagefetch

Fetch any page by the cheapest means that works.

A plain HTTP request is enough for static HTML. Pages that render in JavaScript, or that block anything that looks automated, need a browser. pagefetch starts with the HTTP request and reads what comes back. A bot wall, an error page, or a body too short to be a real page sends it up a tier: first a headless browser, then a headed one. Anything else it returns as it stands.

Features

• Detection — bot walls, throttle pages and soft-404s, spotted in the response body
• Batch mode — many pages in one browser session, so the browser launches once rather than per URL
• Caching — on disk, keyed by URL and content mode, with no TTL
• Self-healing — junk entries are deleted on read and re-fetched, or swept with --clean-cache
• URL allowlist — anything that is not http or https is rejected before a request is made or a browser is launched
• Testable — swap in a FakeFetcher and consuming code needs no network

Quick start

Prerequisites: Python 3.10 or later.

git clone https://github.com/braboj/page-fetcher.git
cd page-fetcher
py -m pip install -e .
py -m pagefetch https://en.wikipedia.org/wiki/Web_scraping

The install is required, not a convenience — the package lives in src/, so it is not importable from a bare checkout. Tier 1 pulls in no dependencies.

That prints the page text to stdout — tens of kilobytes of it, starting:

Web scraping - Wikipedia Jump to content Main menu Main menu move to
sidebar hide Navigation Main page Contents Current events Random article

To use it as a library, import pagefetch from anywhere — the install above is what resolves it, not the working directory:

from pagefetch import NetworkFetcher, FetchOptions, ContentMode

fetcher = NetworkFetcher()
result = fetcher.fetch(
    "https://en.wikipedia.org/wiki/Web_scraping",
    FetchOptions(mode=ContentMode.HTML),
)
if result.ok:
    print(result.tier_used, len(result.content))

That prints the tier that served the page and the size of the body — for this URL, http and a few hundred kilobytes of raw HTML.

Usage

Command line

Fetch one page as text, letting the fetcher pick its own tier:

py -m pagefetch https://en.wikipedia.org/wiki/Web_scraping

Stdout carries the stripped page text. A successful tier 1 fetch says nothing on stderr — the tiers only narrate when they escalate:

[http] Not real content (559 bytes) — escalating
[auto] Skipping js (bot protection), trying headed...

Force one transport instead of letting the fetcher escalate:

py -m pagefetch <url> --http       # plain request only, never escalate
py -m pagefetch <url> --js         # browser that renders JavaScript
py -m pagefetch <url> --headed     # bot bypass, needs a display
py -m pagefetch <url> --headless   # bot bypass, no display needed

Change the output, the wait, or the cache:

py -m pagefetch <url> --format html   # raw HTML instead of stripped text
py -m pagefetch <url> --format text   # stripped text, the default
py -m pagefetch <url> --wait 5000     # extra post-load wait (ms)
py -m pagefetch <url> --no-cache      # refetch, ignoring any cached copy
py -m pagefetch <url> --cache-dir DIR

Fetch many pages in one browser session:

py -m pagefetch --batch urls.txt                    # one URL per line
py -m pagefetch --batch -                           # from stdin
py -m pagefetch url1 url2 url3                      # from arguments
py -m pagefetch --batch urls.txt --output-dir out/  # one file per URL

Purge junk that accumulated in the cache before the current guards existed:

py -m pagefetch --clean-cache
py -m pagefetch --clean-cache --dry-run   # list junk, delete nothing

--dry-run prints the entries it would remove and exits without deleting.

Exit codes

Code	Meaning
0	Every requested URL returned content
1	Nothing came back, or the arguments were rejected
2	A batch returned content for some URLs but not all

A failed fetch writes nothing to stdout, so py -m pagefetch "$url" > page.txt && process page.txt stops rather than processing an empty file. Partial batch failure has its own code because "some pages are missing" and "nothing came back" usually call for different handling.

Library

Inject a PageSource into your own code so it can be faked in tests:

def scrape(source: PageSource, url: str) -> str:
    return source.fetch(url, FetchOptions(mode=ContentMode.HTML)).content


# in tests — no network, no browser:
from pagefetch import FakeFetcher

assert scrape(FakeFetcher({url: "<html>...</html>"}), url) == "<html>...</html>"

Map values are page bodies — the HTML a real fetch would have returned. FakeFetcher derives TEXT mode from them the same way NetworkFetcher does, by stripping tags, so code that reads both modes sees them differ under test as it will in production.

Symbol	Purpose
PageSource	Abstract base — the transport interface
NetworkFetcher	Real fetcher with four-tier escalation
FakeFetcher	Test double backed by a URL→content map
FileCache	On-disk response cache (configurable directory)
FetchOptions	mode, transport, wait_ms, use_cache
FetchResult	url, content, tier_used, ok
ContentMode	TEXT (stripped) / HTML (raw)
Transport	AUTO / HTTP / JS / HEADED / HEADLESS
is_bot_blocked	Pure bot-detection predicate

Point the cache somewhere specific:

from pathlib import Path
from pagefetch import NetworkFetcher, FileCache

fetcher = NetworkFetcher(cache=FileCache(cache_dir=Path("/my/cache")))

Project structure

Path	Purpose
src/pagefetch/	The package — import this
src/pagefetch/source.py	PageSource ABC and the option / result types
src/pagefetch/errors.py	The error contract — what the package raises
src/pagefetch/network.py	NetworkFetcher — the four-tier escalation ladder
src/pagefetch/detection.py	Bot-wall, error-page, and real-content predicates
src/pagefetch/cache.py	FileCache — on-disk cache and junk sweep
src/pagefetch/chrome.py	Chrome / CDP launch helpers for the browser tiers
src/pagefetch/fake.py	FakeFetcher test double
src/pagefetch/__main__.py	CLI entry point — a thin wrapper over the library
tests/	pytest suite, including captured HTML fixtures
examples/	Runnable usage patterns, with their own index
tools/	Repository checks the linter cannot express
docs/arc42/	Architecture documentation, one file per chapter
docs/assets/	Chapter diagrams — draw.io source and PNG export
docs/decisions/	Architecture Decision Records
docs/audits/	360-degree audit reports, one per run
docs/solid-ai-templates/	Quality conventions — a git submodule
docs/ONBOARDING.md	Fresh clone to a passing gate
docs/PLAYBOOK.md	Operational reference for recurring tasks
docs/dev-journal.md	Session log — what changed and why
.github/workflows/	CI and CodeQL pipelines
.vscode/	Editor config mirroring the CI gates — two files
CLAUDE.md	Project rules for AI agents
pyproject.toml	Package metadata and every tool's configuration

Development setup

Clone the repository, install the dependencies and run the test suite:

git clone https://github.com/braboj/page-fetcher.git
cd page-fetcher
py -m pip install -e ".[dev]"
pre-commit install
py -m pytest

To exercise the browser tiers locally, add the optional engines:

py -m pip install -e ".[browsers]"
playwright install chromium

The gate runs from the toolchain, each check runnable on its own:

py -m ruff check .          # lint
py -m ruff format --check . # formatting, including Python in this README
py -m mypy                  # type check
py -m pytest --cov=pagefetch

Beside those are the repository checks in tools/, for conventions no linter expresses — comment layout, code citations, journal order and diagram exports. Each is silent on success. docs/ONBOARDING.md lists them with their arguments.

pre-commit install wires all of it to run on every commit, plus a secret scan. CI repeats every one of them on each pull request, because a hook can be skipped with --no-verify.

Configuration reference

cache_dir is the only configurable value. It resolves in this precedence, highest first:

Source	Type	Default	Notes
--cache-dir DIR	path	unset	CLI flag, one invocation
cache_dir= constructor arg	path	unset	Programmatic use
PAGEFETCH_CACHE_DIR	path	unset	Environment variable
built-in default	path	./.cache/pagefetch	Relative to the working dir

An empty value is an error, not a fallback. --cache-dir "" and PAGEFETCH_CACHE_DIR= both fail naming the source, rather than quietly resolving to the next row down — which is what an unset variable expanded into a wrapper script would otherwise do.

Known limitations

• For research, not bulk scraping — no rate limiting, no backoff, no robots.txt.
• auto checks whether a response failed, not whether it is complete: a page over 10 KB that renders in JavaScript comes back partial. Force --js when completeness matters.
• headed opens a Chrome window, so it cannot run in CI or on a host with no display.
• headless costs ~18-24s, most of it Chrome launch overhead.
• PerimeterX "Press & Hold" blocks every tier.
• Each URL launches its own browser unless you use batch mode.
• The scheme check is an allowlist, not SSRF protection.

Links

• Architecture — the escalation ladder, detection rules, cache behaviour, and the risk register, as arc42 chapters
• Examples — runnable usage patterns, each with the command and the output it produces
• Onboarding — fresh clone to a passing gate
• Playbook — git workflow, adding a tier or a detection pattern, the quality checks, maintenance

Dependencies

Dependency	Tier	Required	License
urllib (stdlib)	http	always	PSF
playwright	js	optional	Apache-2.0
nodriver	headed	optional	AGPL-3.0
seleniumbase	headless	optional	MIT

License

MIT — see LICENSE.

The optional nodriver dependency is AGPL-3.0 and is not covered by this license. It affects you only if you install it and then distribute a network service built on the headed tier.