|
| 1 | +# CLAUDE.md |
| 2 | + |
| 3 | +This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. |
| 4 | + |
| 5 | +## What this repository is |
| 6 | + |
| 7 | +QuantDinger is a self-hosted "AI Trading OS": AI research → strategy code → backtest → paper/live execution → monitoring. This repo contains the **backend only**: Flask API + all worker processes, Compose deployment stacks, ops config (Prometheus/Grafana/Alertmanager), docs, and a standalone MCP server. The web/mobile frontends live in separate private repos (QuantDinger-Vue, QuantDinger-Mobile) and are consumed here as published GHCR images. There is no frontend source in this tree. |
| 8 | + |
| 9 | +Python 3.12 backend (`backend_api_python/`), PostgreSQL 18, two Redis 8 instances, Docker Compose for everything. |
| 10 | + |
| 11 | +## Common commands |
| 12 | + |
| 13 | +All backend commands run from `backend_api_python/` unless noted. |
| 14 | + |
| 15 | +### Development setup |
| 16 | + |
| 17 | +```bash |
| 18 | +cd backend_api_python |
| 19 | +python -m venv .venv && source .venv/bin/activate |
| 20 | +pip install -r requirements-dev.txt |
| 21 | +cp env.example .env # set SECRET_KEY, ADMIN_USER/ADMIN_PASSWORD at minimum |
| 22 | +python run.py # dev server with auto-reload on http://localhost:5000 |
| 23 | +``` |
| 24 | + |
| 25 | +Tests need PostgreSQL + Redis (`DATABASE_URL`, `REDIS_HOST/REDIS_PORT` env) — CI and the compose stack both provide them (`SKIP_STARTUP_HOOKS=1` in CI). Apply migrations first: |
| 26 | + |
| 27 | +```bash |
| 28 | +QD_PROCESS_ROLE=migration python -m app.commands.migrate |
| 29 | +``` |
| 30 | + |
| 31 | +### Tests |
| 32 | + |
| 33 | +```bash |
| 34 | +# Default suite (unit + contract; skips live-exchange and stress markers, and release gates) |
| 35 | +python -m pytest -m "not integration and not stress" --ignore=tests/release_gate -q |
| 36 | + |
| 37 | +# Single test |
| 38 | +python -m pytest tests/test_agent_v1.py::test_whoami_requires_token -q |
| 39 | + |
| 40 | +# Release gates (live-execution safety — run separately) |
| 41 | +python -m pytest tests/release_gate -q |
| 42 | +``` |
| 43 | + |
| 44 | +`integration` = live exchange smoke tests needing real testnet keys; `stress` = long-running synthetic market tests (see `pytest.ini`). |
| 45 | + |
| 46 | +### Lint and guardrails (all run in CI: `.github/workflows/basic-ci.yml`) |
| 47 | + |
| 48 | +```bash |
| 49 | +ruff check app scripts tests # ruff ~0.16, line-length 120, critical rules only (E9/F63/F7/F82) |
| 50 | +python -m compileall -q app scripts tests # syntax check |
| 51 | +python scripts/backend_quality_check.py # legacy-hotspot regression guard (baseline: backend_quality_baseline.json) |
| 52 | +python scripts/check_requirements_lock.py # requirements.lock consistency |
| 53 | +``` |
| 54 | + |
| 55 | +### Repo-level checks (run from repo root) |
| 56 | + |
| 57 | +```bash |
| 58 | +python scripts/check_version.py # VERSION file vs package/artifact versions |
| 59 | +python scripts/check_mojibake.py # text encoding |
| 60 | +python scripts/check_docs.py # docs structure and links |
| 61 | +docker compose -f docker-compose.yml config -q |
| 62 | +docker compose -f docker-compose.yml -f docker-compose.production.yml -f docker-compose.observability.yml config -q |
| 63 | +``` |
| 64 | + |
| 65 | +### Docker stacks |
| 66 | + |
| 67 | +- `docker compose up -d --build` — local/source stack (web :8888, mobile H5 :8889, API :5000; observability not started by default) |
| 68 | +- `-f docker-compose.observability.yml` — adds Prometheus/Grafana/Alertmanager |
| 69 | +- `-f docker-compose.production.yml` — hardened non-root/read-only overlay |
| 70 | +- `docker-compose.ghcr.yml` — prebuilt-image install stack (what `install.sh` deploys) |
| 71 | +- `docker-compose.build.yml` — build frontend from a gitignored `./QuantDinger-Vue/` clone at repo root; pin consumed images via `IMAGE_TAG` / `FRONTEND_TAG` in root `.env` |
| 72 | +- Validate production config: `python backend_api_python/scripts/check_production_config.py --env-file .env --env-file backend_api_python/.env` |
| 73 | + |
| 74 | +### MCP server |
| 75 | + |
| 76 | +```bash |
| 77 | +cd mcp_server |
| 78 | +pip install -e ".[dev]" |
| 79 | +pytest |
| 80 | +``` |
| 81 | + |
| 82 | +Package `quantdinger-mcp` (thin HTTP wrapper over the Agent Gateway). Transport selected by `QUANTDINGER_MCP_TRANSPORT`: `stdio` (default, desktop IDEs), `sse`, or `streamable-http` (remote agents; also set `QUANTDINGER_MCP_HOST/PORT`). |
| 83 | + |
| 84 | +## Architecture: the big picture |
| 85 | + |
| 86 | +### One image, six process roles |
| 87 | + |
| 88 | +The single backend Dockerfile is reused by containers with different `command:` entries plus a `QD_PROCESS_ROLE` env var, validated in `app/runtime/roles.py`: |
| 89 | + |
| 90 | +| Role | Entrypoint | Owns | |
| 91 | +| --- | --- | --- | |
| 92 | +| `migration` | `python -m app.commands.migrate` | Schema migrations, exits before app services start | |
| 93 | +| `api` | Gunicorn (`run.py`) | HTTP, auth, validation, durable command submission | |
| 94 | +| `trading` | `python -m app.commands.trading_worker` | Strategy runtimes, pending orders, broker sessions, reconciliation | |
| 95 | +| `scheduler` | `python -m app.commands.scheduler` | Portfolio/deployment/payment/signal schedules | |
| 96 | +| `celery` (worker + beat) | `celery -A app.celery_app:celery_app worker/beat` | Finite retryable jobs (AI, backtest, experiment, report, maintenance), queues `jobs,ai,maintenance` | |
| 97 | + |
| 98 | +**Ownership rules (hard invariant, verified by `tests/release_gate/test_live_execution_release_gate.py`; detailed in `docs/architecture/PROCESS_ROLES_AND_TASKS.md`):** HTTP routes validate and delegate — they must never own trading loops, exchange-specific behavior, or large DB workflows. Long-lived trading loops belong to the trading worker (commands flow: API route → PostgreSQL command record → trading-worker → strategy runtime/broker adapter). Finite retryable work belongs to Celery via the **jobs Redis**, which is a separate instance from the disposable **cache Redis** — never use cache Redis as the Celery broker. |
| 99 | + |
| 100 | +### Module map (`backend_api_python/app/`) |
| 101 | + |
| 102 | +- `routes/` — thin HTTP route facades (human API + `routes/agent_v1/` = Agent Gateway under `/api/agent/v1`) |
| 103 | +- `services/` — domain workflows: `strategy_v2/` (versioned strategy contracts), `strategy_runtime/` (signals, intents, execution, state), `live_trading/` (normalized crypto exchange adapters: Binance, OKX, Bybit, Bitget, Gate, HTX + factory), `ibkr_trading/`, `alpaca_trading/`, `backtest_engine/` |
| 104 | +- `data_sources/` — raw market-data source adapters (CCXT, yfinance, ...); `data_providers/` — aggregated dashboard/macro/news/sentiment providers that fan out across sources |
| 105 | +- `markets/` — symbol/catalog normalization |
| 106 | +- `tasks/` — Celery jobs (registered in `celery_app.py`); `workers/`, `commands/`, `runtime/` — process shells and role/ownership helpers |
| 107 | +- `openapi/` — flask-smorest schemas/blueprints; `observability/` — metrics, request IDs, JSON logs |
| 108 | +- `migrations/` — **raw date-prefixed SQL files**, applied in order by the migration role |
| 109 | + |
| 110 | +### OpenAPI is the contract SSOT — CI enforces it |
| 111 | + |
| 112 | +- Human web API: `docs/api/openapi.yaml`, regenerated with `python scripts/export_openapi.py` |
| 113 | +- Agent Gateway: `docs/agent/agent-openapi.json`, updated by hand |
| 114 | +- Any route change must update the corresponding artifact; `.github/workflows/openapi-ci.yml` runs Spectral lint, export diff, and oasdiff breaking-change checks. Read `docs/architecture/API_CONVENTIONS.md` before adding public endpoints. |
| 115 | + |
| 116 | +### Agent Gateway and MCP |
| 117 | + |
| 118 | +Auth: `app/utils/agent_auth.py` `@agent_required(scope=...)`; tokens hashed at rest in `qd_agent_tokens` (never log raw tokens); every call (success and denial) audited to `qd_agent_audit`; rate-limited. Async jobs go through `app/utils/agent_jobs.py` with SSE progress streaming (`GET /jobs/{id}/stream`). **Trading is paper-only by default** — live execution requires token `paper_only=false` AND server env `AGENT_LIVE_TRADING_ENABLED=true` AND operator limits/allowlists. The MCP server exposes R+W+B endpoints only (no trading); add an MCP tool only after the capability exists as a REST endpoint. Read `docs/agent/AGENT_ENVIRONMENT_DESIGN.md` and `docs/agent/AI_INTEGRATION_DESIGN.md` before changing any agent-facing surface. |
| 119 | + |
| 120 | +## Conventions |
| 121 | + |
| 122 | +- Code comments, docstrings, and log messages in **English** (docs/agent must be English-only). |
| 123 | +- Branch naming: `fix/`, `feat/`, `docs/`, `chore/` prefixes. |
| 124 | +- Keep routes thin: validate → call service → return JSON. Put new behavior in focused sibling modules rather than growing legacy hotspot files. |
| 125 | +- Security red lines (do not weaken without an explicit request): agent live-trading gates, credential encryption (`CREDENTIAL_ENCRYPTION_KEY` — stable, separate from `SECRET_KEY`), hashed tokens, non-root production runtime, loopback-only published ports. |
| 126 | +- Never commit secrets or production `.env` files; use `env.example` patterns. Settings UI writes runtime config to `/app/.env` (host `backend.env` in GHCR stack, `backend_api_python/.env` in source deployments) — that file is mode 600, owned by UID 10001. |
| 127 | +- Version changes touch `VERSION` (root) — `scripts/check_version.py` verifies consistency across package files; release tags are `vX.Y.Z`. |
| 128 | + |
| 129 | +## Where to read deeper |
| 130 | + |
| 131 | +Start from the docs index `docs/README.md`. The docs are the authoritative design record: |
| 132 | + |
| 133 | +- `docs/architecture/` — ARCHITECTURE.md, MODULE_BOUNDARIES.md, PROCESS_ROLES_AND_TASKS.md, CONCURRENCY_MODEL.md, API_CONVENTIONS.md, EXTENSION_GUIDE.md |
| 134 | +- `docs/trading/` — STRATEGY_DEV_GUIDE.md (Strategy API V2), INDICATOR_DEV_GUIDE.md |
| 135 | +- `docs/agent/` — AGENT_ENVIRONMENT_DESIGN.md, AI_INTEGRATION_DESIGN.md, MCP_SETUP.md, agent-openapi.json |
| 136 | +- `docs/deployment/` — INSTALL_TROUBLESHOOTING.md, CLOUD_DEPLOYMENT_EN.md, PRODUCTION_HARDENING.md, OBSERVABILITY.md |
| 137 | +- Root README.md — user-facing overview, install paths, and a "Where changes belong" table for routing edits to the right module |
0 commit comments