Skip to content

.rules.md

Latest commit

 

History

History
70 lines (42 loc) · 3.91 KB

.rules.md

File metadata and controls

70 lines (42 loc) · 3.91 KB
Raw

Coding guidelines

This file provides guidance to programming agents when working with code in this repository.

Crawlee for Python (crawlee on PyPI) is a web scraping and browser automation framework.

Read CONTRIBUTING.md first. It covers setup, tasks, tests, code style, documentation, and commits, and all of it applies to agents too.

@CONTRIBUTING.md

Pydantic models

Don't introduce a shared Pydantic base model to centralize model_config. Each model inherits pydantic.BaseModel and declares model_config = ConfigDict(...) inline, which also keeps ruff's flake8-type-checking and RUF012 checks working, since they only recognize a literal pydantic.BaseModel base.

Architecture

Crawler hierarchy

BasicCrawler[TCrawlingContext, TStatisticsState]
├── AbstractHttpCrawler  →  HttpCrawler, BeautifulSoupCrawler, ParselCrawler, FileDownloadCrawler, PydanticAiCrawler
├── PlaywrightCrawler    →  StagehandCrawler
└── AdaptivePlaywrightCrawler (runs a static HTTP sub-crawler and a PlaywrightCrawler, picks per request)

BasicCrawler (crawlers/_basic/) owns the request lifecycle, autoscaling pool, retries, sessions, and router dispatch. AbstractHttpCrawler (crawlers/_abstract_http/) adds HTTP client integration, response parsing, and pre-navigation hooks.

Context pipeline

Contexts are progressively enhanced through ContextPipeline middleware:

BasicCrawlingContext → HttpCrawlingContext → ParsedHttpCrawlingContext → BeautifulSoupCrawlingContext

Each middleware is an async generator that wraps the next handler, enabling setup/teardown around request processing.

Storage layer

Three-tier design:

  • High-level: Dataset, KeyValueStore, RequestQueue in src/crawlee/storages/
  • Storage clients (src/crawlee/storage_clients/): FileSystemStorageClient (default), MemoryStorageClient, SqlStorageClient, RedisStorageClient
  • Instance caching: StorageInstanceManager is a global singleton that caches storage instances by ID/name

Service locator

src/crawlee/_service_locator.py is a global singleton managing Configuration, EventManager, StorageClient, and StorageInstanceManager. Prevents double-initialization with ServiceConflictError.

HTTP clients

Pluggable via HttpClient interface in src/crawlee/http_clients/:

  • ImpitHttpClient (default), HttpxHttpClient (wraps the httpx2 package), CurlImpersonateHttpClient
  • Each provides crawl() (for crawler pipeline) and send_request() (for in-handler use)

Request model

Request (src/crawlee/_request.py) uses unique_key for deduplication. Lifecycle (RequestState): UNPROCESSED → BEFORE_NAV → AFTER_NAV → REQUEST_HANDLER → DONE, with ERROR_HANDLER → ERROR on failure and SKIPPED for requests the crawler skips (currently only when disallowed by robots.txt). Crawlee-specific metadata stored in user_data['__crawlee'].

Private modules used by the Apify SDK

apify-sdk-python imports from several private crawlee modules, so changes there can break it even when nothing in crawlee breaks:

  • crawlee._log_config: the SDK calls configure_logger(..., remove_old_handlers=True) and uses CrawleeLogFormatter and get_configured_log_level. Keep their signatures and behavior stable, even though crawlee itself never passes remove_old_handlers=True.
  • crawlee._types, crawlee._request, crawlee._consts, and helpers under crawlee._utils (crypto, urls, models, file, requests, byte_size, log, try_import).
  • crawlee.events._types and crawlee.events._event_manager.
  • crawlee.storage_clients._base and crawlee.storage_clients._file_system, whose file-system clients the SDK subclasses, so their internals count as an interface too.

Before renaming, removing, or changing the behavior of anything in these modules, search apify-sdk-python for its use.