|
2 | 2 |
|
3 | 3 | This file provides guidance to programming agents when working with code in this repository. |
4 | 4 |
|
5 | | -## Project Overview |
| 5 | +Python client for the [Apify API](https://docs.apify.com/api/v2), published on PyPI as `apify-client`. `ApifyClient` (sync) and `ApifyClientAsync` (async) are the entry points. |
6 | 6 |
|
7 | | -Python client for the [Apify API](https://docs.apify.com/api/v2). Provides `ApifyClient` (sync) and `ApifyClientAsync` (async) classes to interact with the Apify platform. Published on PyPI as `apify-client`. |
| 7 | +Read [CONTRIBUTING.md](CONTRIBUTING.md) first. It covers setup, tasks, tests, code style, generated models, documentation, and commits, and all of it applies to agents too. |
8 | 8 |
|
9 | | -## Development Commands |
10 | | - |
11 | | -Uses [uv](https://docs.astral.sh/uv/) for project management and [Poe the Poet](https://poethepoet.natn.io/) as task runner. Requires Python 3.11+. |
12 | | - |
13 | | -```bash |
14 | | -uv run poe install-dev # Install dev deps + git hooks |
15 | | -uv run poe check-code # Run all checks (lint, type-check, unit-tests, docstring check) |
16 | | -uv run poe lint # Ruff format check + ruff check |
17 | | -uv run poe format # Auto-fix lint issues and format |
18 | | -uv run poe type-check # Run ty type checker |
19 | | -uv run poe unit-tests # Run unit tests |
20 | | -uv run poe check-docstrings # Verify async docstrings match sync |
21 | | -uv run poe fix-docstrings # Auto-fix async docstrings |
22 | | -uv run poe generate-models # Regenerate _models.py, _typeddicts.py, _literals.py from the published spec |
23 | | -uv run poe generate-models-from-file <path> # Regenerate from a local OpenAPI spec file |
24 | | - |
25 | | -# Run a single test |
26 | | -uv run pytest tests/unit/test_file.py |
27 | | -uv run pytest tests/unit/test_file.py::test_name |
28 | | -``` |
29 | | - |
30 | | -Integration tests require `APIFY_TEST_USER_API_TOKEN` and `APIFY_TEST_USER_2_API_TOKEN` env vars. |
| 9 | +@CONTRIBUTING.md |
31 | 10 |
|
32 | 11 | ## Architecture |
33 | 12 |
|
34 | | -### Client Hierarchy |
| 13 | +### Client hierarchy |
35 | 14 |
|
36 | | -`ApifyClient`/`ApifyClientAsync` are the entry points. They provide methods that return resource sub-clients: |
| 15 | +Entry points return resource sub-clients, which can create child clients (e.g. `ActorClient.builds()` -> `BuildCollectionClient`): |
37 | 16 |
|
38 | 17 | ```text |
39 | 18 | ApifyClient |
40 | | -├── .actor(id) → ActorClient (single resource operations) |
| 19 | +├── .actor(id) → ActorClient (single resource operations) |
41 | 20 | ├── .actors() → ActorCollectionClient (list/create) |
42 | | -├── .dataset(id) → DatasetClient |
43 | | -├── .datasets() → DatasetCollectionClient |
44 | 21 | ├── .run(id) → RunClient |
45 | | -├── .runs() → RunCollectionClient |
46 | | -└── ... (schedules, tasks, webhooks, key-value stores, request queues, etc.) |
| 22 | +└── ... (datasets, schedules, tasks, webhooks, key-value stores, request queues, etc.) |
47 | 23 | ``` |
48 | 24 |
|
49 | | -Each resource client can create child clients (e.g., `ActorClient.builds()` → `BuildCollectionClient`). |
50 | | - |
51 | | -### Sync/Async Symmetry |
52 | | - |
53 | | -Every resource client has both sync and async variants in the **same file**. For example, `actor.py` contains both `ActorClient` and `ActorClientAsync`. Both share the same interface — async versions use `async/await`. |
54 | | - |
55 | | -Docstrings are written on sync clients and **automatically copied** to async clients via `scripts/check_docstrings.py`. When modifying docstrings, edit only the sync client and run `uv run poe fix-docstrings`. |
| 25 | +`ClientRegistry` / `ClientRegistryAsync` (`_client_registry.py`) hold the resource client classes and are passed to every client, so clients can create child clients without importing them (avoids circular imports). All resource clients derive from `ResourceClientBase` (`_resource_clients/_resource_client.py`), whose `WithLogDetailsClient` metaclass wraps public methods for structured logging. |
56 | 26 |
|
57 | | -### Dependency Injection via ClientRegistry |
| 27 | +### Sync/async symmetry |
58 | 28 |
|
59 | | -`ClientRegistry`/`ClientRegistryAsync` (in `_client_registry.py`) holds references to all resource client classes and is passed to each client. This avoids circular imports and lets resource clients create child clients without directly importing them. |
| 29 | +Every resource client in `src/apify_client/_resource_clients/` has its sync and async variant in the **same file** (e.g. `ActorClient` and `ActorClientAsync` in `actor.py`) with the same interface. Docstrings are written on the sync client only (see [Documentation](CONTRIBUTING.md#documentation)). |
60 | 30 |
|
61 | | -### HTTP Client Abstraction |
| 31 | +### HTTP layer |
62 | 32 |
|
63 | | -- `HttpClient`/`HttpClientAsync` — base classes in `http_clients/_base.py` holding the shared request pipeline (retries, |
64 | | - timeouts, API errors); transports implement the `send_request`, error-classification, and lifecycle hooks |
65 | | -- `ImpitHttpClient`/`ImpitHttpClientAsync` — default implementation (Rust-based Impit) |
66 | | -- `Httpx2HttpClient`/`Httpx2HttpClientAsync` — built-in alternative behind the `httpx2` optional extra |
67 | | -- `HttpResponse` — Protocol (not a concrete class) for response objects |
68 | | -- Users can plug in custom HTTP clients via `ApifyClient.with_custom_http_client()` |
| 33 | +`HttpClient` / `HttpClientAsync` (`http_clients/_base.py`) hold the shared request pipeline (retries, timeouts, API errors); transports (`Impit*` default, `Httpx2*` behind the `httpx2` extra, or a custom one via `ApifyClient.with_custom_http_client()`) implement sending, error classification, and lifecycle hooks. Request-body compression lives in `http_compressors/`, selected by the `compression` constructor argument. |
69 | 34 |
|
70 | | -### Base Classes |
| 35 | +### Generated models |
71 | 36 |
|
72 | | -- `ResourceClientBase` — shared URL building, parameter handling, metaclass-based logging |
73 | | -- `ResourceClient` / `ResourceClientAsync` — sync/async base classes with HTTP call methods |
74 | | -- `WithLogDetailsClient` metaclass — auto-wraps public methods for structured logging |
| 37 | +`_models.py`, `_typeddicts.py`, and `_literals.py` are **generated**; never edit them by hand (see [Generated models](CONTRIBUTING.md#generated-models)). Each input-side TypedDict ships in a snake_case and a camelCase variant (`RequestResourceDict`, `RequestResourceCamelDict`); the camel variants are synthesized in the post-processing script, and resource-client signatures accept either casing or the Pydantic model. |
75 | 38 |
|
76 | | -### Data Models |
| 39 | +### Checking API response shapes |
77 | 40 |
|
78 | | -`src/apify_client/_models.py` and `src/apify_client/_typeddicts.py` are **auto-generated** — do not edit them manually. Every Pydantic model and TypedDict comes from the OpenAPI spec. |
| 41 | +The OpenAPI spec can lag behind the API and omit fields, and a live response only shows the fields that happen to be set. If the private `apify-core` repository is available locally, use it to confirm which fields an endpoint returns: |
79 | 42 |
|
80 | | -Each input-side TypedDict ships in two casings: snake_case (`RequestResourceDict`) and camelCase (`RequestResourceCamelDict`). The camel variants are synthesized in `scripts/postprocess_generated_models.py` from the Pydantic `Field(alias=...)` map; resource-client signatures union both with the Pydantic model so users can pass either casing without losing type-checker support. |
81 | | - |
82 | | -- Generated by `datamodel-code-generator` from the OpenAPI spec at `https://docs.apify.com/api/openapi.json` (config in `pyproject.toml` under `[tool.datamodel-codegen]`, aliases in `datamodel_codegen_aliases.json`) |
83 | | -- After generation, `scripts/postprocess_generated_models.py` is run to apply additional fixes |
84 | | -- To regenerate locally: |
85 | | - - From the live published spec: `uv run poe generate-models` |
86 | | - - From a local spec file: `uv run poe generate-models-from-file path/to/openapi.json` |
87 | | -- `scripts/openapi_spec.py` downloads the spec once into git-ignored `tmp/openapi.json` — both codegen passes read that copy, so a spec redeployed mid-run can't leave the generated files describing two different inputs — then records its `info.version` in `pyproject.toml` under `[tool.apify.openapi-spec]` |
88 | | -- That version is a **coarse marker, not a content identity**: the spec is served latest-only, and apify-docs bumps `components/version.yaml` in a follow-up `[skip ci]` commit, so a deploy can publish new content under the old stamp. A moved value proves the spec changed; an unchanged value proves nothing |
89 | | -- Recording is unconditional; the nightly workflow keeps it honest by committing nothing unless the models changed. A local `generate-models` that only moves that line should be dropped, not committed alone. `generate-models-from-file` leaves the line alone, so models from a candidate spec must not be committed |
90 | | -- In CI, the `Regenerate models` workflow (`on_schedule_regenerate_models.yaml`) runs nightly at 02:00 UTC and opens a PR when the models change. It always generates on master and rebuilds its `ci/regenerate-models` branch from master instead of appending, so the PR diff is always "current spec vs current master". Retitle the `chore:` PR to `fix:`/`feat:` when the diff is user-facing, so it lands in the changelog and triggers a release |
91 | | -- The gate is the **generated models**, not the spec version — which moves on rebuilds regardless of client impact, and doesn't reliably move when the content does |
92 | | -- Manual regeneration is also possible from the GitHub Actions UI (`Regenerate models` workflow) |
93 | | - |
94 | | -## Code Conventions |
95 | | - |
96 | | -- **Line length**: 120 characters |
97 | | -- **Linting/formatting**: Ruff with nearly all rules enabled (see `pyproject.toml`) |
98 | | -- **Type checking**: ty (Astral's type checker) |
99 | | -- **Docstrings**: Google style format, single backticks for inline code references (`` `ApifyClient` `` not ``` ``ApifyClient`` ```) |
100 | | -- **Imports**: `from __future__ import annotations` used throughout |
101 | | -- **Commits**: [Conventional Commits](https://www.conventionalcommits.org/) format. Choose the type based on *what* changed, not just *why*: |
102 | | - - `feat:` / `fix:` / `perf:` / `refactor:` / `style:` — **source code only**; these trigger a release and appear in the changelog |
103 | | - - `test:` — test additions or changes (no release triggered) |
104 | | - - `docs:` — documentation changes; also triggers a doc release on master |
105 | | - - `ci:` — CI/workflow changes |
106 | | - - `chore:` — dependency bumps, tooling, generated files (e.g. model regeneration PRs), and other housekeeping |
107 | | - - `build:` — build system changes |
108 | | - |
109 | | -## Testing |
110 | | - |
111 | | -- **Unit tests** (`tests/unit/`): Use `pytest-httpserver` to mock HTTP. No network required. |
112 | | -- **Integration tests** (`tests/integration/`): Hit the live Apify API. |
113 | | -- **Async**: `asyncio_mode = "auto"` — async tests run automatically without markers. |
114 | | -- **Parallelism**: Tests run in parallel via `pytest-xdist`. |
115 | | - |
116 | | -Unit test pattern: |
117 | | - |
118 | | -```python |
119 | | -def test_example(httpserver: HTTPServer) -> None: |
120 | | - httpserver.expect_request('/v2/endpoint').respond_with_json({'data': ...}) |
121 | | - client = ApifyClient(token='test', api_url=httpserver.url_for('/').removesuffix('/')) |
122 | | - # assert client behavior |
123 | | -``` |
| 43 | +- A single resource's fields are in `PUBLISHED_*_FIELDS` in `src/packages/<area>-server/src/<resource>/<resource>.server.ts`. |
| 44 | +- A list endpoint has its own, usually smaller, `API_PUBLISHED_*_LIST_FIELDS` in `src/api/src/routes/<resource>/<resource>_list.ts`, and often builds each item inline. Don't assume a list item has the same fields as the single resource. |
0 commit comments