|
| 1 | +# AGENTS.md |
| 2 | + |
| 3 | +Instructions for AI coding agents (Claude Code, Codex, Cursor, Copilot agents, …) |
| 4 | +working on **ScrapeGraphAI**. Human contributors should read |
| 5 | +[CONTRIBUTING.md](CONTRIBUTING.md); everything here is in addition to it. |
| 6 | + |
| 7 | +--- |
| 8 | + |
| 9 | +## 1. Golden rule: everything goes to `pre/beta` |
| 10 | + |
| 11 | +**`main` is never written to directly. All work is based on and merged into `pre/beta`.** |
| 12 | + |
| 13 | +`pre/beta` is the prerelease branch: pushes to it publish a `beta` prerelease via |
| 14 | +semantic-release (see `.releaserc.yml`). `main` only receives releases when a |
| 15 | +maintainer promotes `pre/beta`. |
| 16 | + |
| 17 | +```bash |
| 18 | +# 1. always start from an up-to-date pre/beta |
| 19 | +git fetch origin |
| 20 | +git checkout -b feat/my-change origin/pre/beta |
| 21 | + |
| 22 | +# 2. commit your work |
| 23 | +git add <only the files you touched> |
| 24 | +git commit -m "feat(nodes): add X" |
| 25 | + |
| 26 | +# 3. push and open the PR against pre/beta |
| 27 | +git push -u origin feat/my-change |
| 28 | +gh pr create --base pre/beta --title "feat(nodes): add X" --body "..." |
| 29 | +``` |
| 30 | + |
| 31 | +Checklist before you commit: |
| 32 | + |
| 33 | +- [ ] The branch is based on `origin/pre/beta` (`git merge-base --is-ancestor origin/pre/beta HEAD`). |
| 34 | +- [ ] The PR base is `pre/beta`, **not** `main`. |
| 35 | +- [ ] No commits directly on `main` or `pre/beta`, no force-push to either. |
| 36 | +- [ ] One logical change per branch/PR. |
| 37 | + |
| 38 | +If a task genuinely requires targeting `main` (e.g. a hotfix on a released |
| 39 | +version), stop and ask a maintainer first. |
| 40 | + |
| 41 | +## 2. Environment setup |
| 42 | + |
| 43 | +Python `>=3.12`, dependencies managed with [uv](https://docs.astral.sh/uv/): |
| 44 | + |
| 45 | +```bash |
| 46 | +uv sync # create the venv and install deps |
| 47 | +uv run pre-commit install # install the git hooks |
| 48 | +``` |
| 49 | + |
| 50 | +Never hand-edit `uv.lock`; regenerate it with `uv lock` / `uv sync` and commit |
| 51 | +the result only when you actually changed dependencies in `pyproject.toml`. |
| 52 | + |
| 53 | +## 3. Checks to run before pushing |
| 54 | + |
| 55 | +```bash |
| 56 | +make lint # ruff + black --check + isort --check-only |
| 57 | +make type-check # mypy (strict) |
| 58 | +make test # pytest with coverage |
| 59 | +make pre-commit # run all hooks on all files |
| 60 | +``` |
| 61 | + |
| 62 | +Run at least `make lint` and the tests covering what you touched. Report the |
| 63 | +real result: if something fails or you skipped a step, say so in the PR |
| 64 | +description instead of implying a clean run. |
| 65 | + |
| 66 | +Style: PEP 8 + Google Python docstrings, `black` formatting, line length 88. |
| 67 | +Match the conventions of the surrounding file rather than introducing new ones. |
| 68 | + |
| 69 | +## 4. Commit messages |
| 70 | + |
| 71 | +Commits are parsed by semantic-release (Conventional Commits, `conventionalcommits` |
| 72 | +preset), so the message decides the next version number. Use: |
| 73 | + |
| 74 | +``` |
| 75 | +feat: ✨ new feature -> minor bump |
| 76 | +fix: 🐛 bug fix -> patch bump |
| 77 | +docs: 📚 documentation |
| 78 | +style: 💅 formatting only |
| 79 | +refactor: ♻️ no behaviour change |
| 80 | +perf: ⚡ performance |
| 81 | +test: 🧪 tests |
| 82 | +build: 📦 build system / deps |
| 83 | +ci: 🤖 CI configuration |
| 84 | +chore: 🧹 everything else |
| 85 | +``` |
| 86 | + |
| 87 | +Format: `type(optional-scope): imperative summary`, optional body, and |
| 88 | +`BREAKING CHANGE:` in the footer for incompatible changes. Reference issues with |
| 89 | +`Fixes #123`. |
| 90 | + |
| 91 | +## 5. Files agents must not touch |
| 92 | + |
| 93 | +- `CHANGELOG.md` and the `version` field in `pyproject.toml` — owned by |
| 94 | + semantic-release; editing them by hand breaks releases. |
| 95 | +- Git tags and release notes on GitHub. |
| 96 | +- `.github/workflows/*` — only when the task is explicitly about CI. |
| 97 | +- Anything under `htmlcov/`, `coverage.xml`, `.pytest_cache/`, `__pycache__/`: |
| 98 | + build artifacts, never commit them. |
| 99 | + |
| 100 | +Also: never commit secrets. API keys go in a local `.env` (git-ignored) and are |
| 101 | +read via `os.getenv`; examples and tests must use placeholders such as |
| 102 | +`OPENAI_APIKEY` from the environment. |
| 103 | + |
| 104 | +## 6. Repository layout |
| 105 | + |
| 106 | +``` |
| 107 | +scrapegraphai/ |
| 108 | +├── graphs/ # pipelines (SmartScraperGraph, SearchGraph, …) |
| 109 | +├── nodes/ # single graph steps (FetchNode, ParseNode, GenerateAnswerNode, …) |
| 110 | +├── models/ # LLM wrappers and token/model metadata |
| 111 | +├── docloaders/ # loaders (ChromiumLoader, …) |
| 112 | +├── prompts/ # prompt templates |
| 113 | +├── helpers/ # shared constants and schemas |
| 114 | +├── integrations/ # third-party / managed-API integrations |
| 115 | +└── utils/ # utilities (html cleanup, tokenization, …) |
| 116 | +examples/ # runnable usage examples, one folder per graph |
| 117 | +tests/ # pytest suite, mirrors the package layout |
| 118 | +docs/ # documentation sources |
| 119 | +``` |
| 120 | + |
| 121 | +When adding a node or graph, register it in the corresponding `__init__.py` and |
| 122 | +add a test under `tests/` next to the existing ones for that layer. New |
| 123 | +user-facing features need an entry in `examples/` and, when they change public |
| 124 | +behaviour, a docs update. |
| 125 | + |
| 126 | +## 7. Working style expected from agents |
| 127 | + |
| 128 | +- Prefer small, reviewable diffs; do not reformat or "clean up" untouched files. |
| 129 | +- Do not add dependencies unless the task requires it — say why in the PR. |
| 130 | +- Write all commits, PR titles/bodies, issue comments, code comments and |
| 131 | + docstrings **in English**. |
| 132 | +- Do not delete or rewrite existing tests to make a change pass. |
| 133 | +- If a test is already failing on `pre/beta`, mention it rather than silently |
| 134 | + fixing unrelated things in the same PR. |
| 135 | +- Never commit other people's in-progress work: check `git status` and stage |
| 136 | + only the files belonging to your change. |
0 commit comments