diff --git a/.claude/CLAUDE.md b/.claude/CLAUDE.md index 2575a1f99e..e2b225d23e 100644 --- a/.claude/CLAUDE.md +++ b/.claude/CLAUDE.md @@ -89,10 +89,26 @@ make mockgen # Regenerate only mocks (after changing interfaces that h - Run `make proto-all` after any `.proto` file change — this regenerates Go bindings and OpenAPI/Swagger specs. - Run `make mockgen` after changing any Go interface that has generated mocks in a `mock/` directory. - The generated files are committed to the repo. `protoc` v23.4 must be on PATH ahead of any other version for the version header in `.pb.go` files to stay at `v4.23.4`. +- Prefer running generation inside the dev container via the `devcontainer-generate` skill — the container guarantees protoc v23.4, while a host version mismatch churns every generated file. ### Local development -**Docker Compose (recommended for most development):** +**Dev container + Minikube (preferred):** + +Development actions — proto generation, builds, deploys, kubectl/helm — should run **inside the Bucketeer dev container**, where tool versions are guaranteed (protoc v23.4, go-tools, minikube/helm/kubectl). Check for a running dev container first before running these on the host. Use the project skills: + +- `devcontainer-run` — detect the running dev container (local devcontainer or Codespace) and run any command inside it: `bash .claude/skills/devcontainer-run/scripts/exec.sh status` +- `devcontainer-generate` — proto/mock generation inside the container +- `devcontainer-deploy` — deploy to the minikube cluster inside the container + +```bash +make start-minikube # Inside the dev container. Always use this, not `minikube start` directly +make deploy-bucketeer # Deploy all Helm charts to minikube +``` + +Never run kubectl/helm bare on the host — the host kubectl context may point at a real GKE cluster, not minikube. + +**Docker Compose (host-based alternative):** ```bash make docker-compose-up # Start all services make docker-compose-status # Check status @@ -107,12 +123,6 @@ Add to `/etc/hosts`: 127.0.0.1 api-gateway.bucketeer.io ``` -**Minikube (Kubernetes-based):** -```bash -make start-minikube # Always use this, not `minikube start` directly -make deploy-bucketeer # Deploy all Helm charts -``` - ### Database migrations ```bash make migration-validate # Validate migration files with Atlas diff --git a/.claude/skills/devcontainer-deploy/SKILL.md b/.claude/skills/devcontainer-deploy/SKILL.md new file mode 100644 index 0000000000..e43bb9c7f0 --- /dev/null +++ b/.claude/skills/devcontainer-deploy/SKILL.md @@ -0,0 +1,95 @@ +--- +name: devcontainer-deploy +description: >- + Build and deploy Bucketeer to the minikube cluster inside the dev container, + or redeploy/restart a single service there. Use this whenever the user wants + to deploy locally, run "make deploy-bucketeer", start minikube, get their + code changes running in the dev cluster, restart a crashing pod, or says + "devcontainer-deploy", "deploy to minikube", "redeploy the backend". Also use it to + check deployment health (pods not ready, gateway not responding) in the dev + container environment. +--- + +# devcontainer-deploy — deploy Bucketeer inside the dev container + +Deployment target is the minikube cluster *inside* the dev container (helm +charts in `manifests/`), not the host docker-compose stack. (Human-facing +docs for these flows: `DEVELOPMENT.md` § "Deploy Bucketeer".) All commands go +through the devcontainer-run wrapper (see the `devcontainer-run` skill): + +```bash +DEVC="bash .claude/skills/devcontainer-run/scripts/exec.sh" +``` + +## 1. Preflight + +```bash +$DEVC status +``` + +- `dockerd` not running → start it (command is in the status output) — image + builds need it. +- `minikube` not running → `$DEVC 'make start-minikube'`. Never `minikube start` + directly. If minikube IS already running, skip this: the target intentionally + exits 1 with "minikube is already running" — that is not an error to fix. + +## 2. Full deploy + +```bash +$DEVC 'make deploy-bucketeer' +``` + +What it does, so failures are diagnosable: uninstalls the existing `bucketeer` +helm release → regenerates cert/token/oauth secrets → builds all Go binaries → +builds docker images with `TAG=localenv` → loads them into minikube → helm +install/upgrade `localenv` (MySQL, Redis, Pub/Sub emulator, optionally +Postgres/BigQuery emulator) → helm install `bucketeer` with +`manifests/bucketeer/values.dev.yaml`. + +This takes many minutes. Run it with a 600000 timeout or `run_in_background` +and monitor. Postgres/BigQuery enablement is auto-detected from +`dataWarehouse` in `values.dev.yaml` — don't set it manually, but remember the +invariant: `web` and `subscriber` must use the same event store, so data +warehouse changes belong in `values.dev.yaml`, not ad-hoc helm flags. + +## 3. Single service — faster than a full deploy + +For a code change to one service (e.g. backend): + +```bash +$DEVC 'make build-go-embed && TAG=localenv make build-docker-images && TAG=localenv make minikube-load-images' +$DEVC 'kubectl --context minikube rollout restart deployment && kubectl --context minikube rollout status deployment ' +``` + +The Bucketeer deployments are `api`, `web`, `batch-server`, and `subscriber` +(confirm with `$DEVC 'kubectl --context minikube get deployments'`). Chart-level +changes instead: + +```bash +$DEVC 'helm upgrade bucketeer manifests/bucketeer/ --kube-context minikube --values manifests/bucketeer/values.dev.yaml' +``` + +Always go through `$DEVC` and always name the context: a bare `helm`/`kubectl` +runs against whatever context is currently active, which on the host is often a +real cluster. `$DEVC status` warns when the active context is not `minikube`. + +## 4. Verify + +```bash +$DEVC 'kubectl --context minikube get pods' # everything Running/Completed, restarts not climbing +$DEVC 'curl -sk https://api-gateway.bucketeer.io/health' # must run INSIDE the container +``` + +The `*.bucketeer.io` hosts entries live in the container's `/etc/hosts` +(pointed at `minikube ip`) — curl from the host proves nothing. For a failing +pod: `$DEVC 'kubectl --context minikube logs deploy/ --tail=100'` and +`$DEVC 'kubectl --context minikube describe pod '`. + +## Related dev-cluster chores + +- Bootstrap e2e accounts (after a fresh deploy, before e2e tests): + `$DEVC 'make create-dev-container-e2e-accounts'` +- Wipe e2e data: `$DEVC 'make delete-dev-container-mysql-data'` (or the + `-postgres-` variant). These are destructive — confirm with the user first. +- MySQL from inside the container: host `$(minikube ip)`, port 32000, + user/pass `bucketeer`, db `bucketeer`. diff --git a/.claude/skills/devcontainer-generate/SKILL.md b/.claude/skills/devcontainer-generate/SKILL.md new file mode 100644 index 0000000000..bcd7a0a622 --- /dev/null +++ b/.claude/skills/devcontainer-generate/SKILL.md @@ -0,0 +1,65 @@ +--- +name: devcontainer-generate +description: >- + Regenerate Bucketeer protobuf Go bindings, OpenAPI/Swagger specs, and gomock + files inside the dev container, where protoc is guaranteed to be exactly + v23.4. Use this whenever a .proto file changed, generated *.pb.go / + *.pb.gw.go / swagger files need regenerating, a mocked Go interface changed + (mockgen), or the user says "generate proto", "regen protos", "make + proto-all", "make mockgen", or "devcontainer-generate". Prefer this over running + protoc or make proto-all on the host — a host protoc version mismatch + silently rewrites every generated file's header. +--- + +# devcontainer-generate — code generation inside the dev container + +Generated files are committed to the repo, and their headers record the protoc +version (`protoc v4.23.4`). The dev container ships exactly protoc 23.4, so +generation must happen there; a different host protoc churns every `.pb.go` +file and the PR becomes unreviewable. (Human-facing background: +`DEVELOPMENT.md` § "Working with the dev container from the host".) + +All commands below go through the devcontainer-run wrapper (see the `devcontainer-run` skill for how +detection works): + +```bash +DEVC="bash .claude/skills/devcontainer-run/scripts/exec.sh" +``` + +## 1. Pick the right target + +| What changed | Target | +|---|---| +| `.proto` files | `make proto-all` | +| A Go interface that has generated mocks in a `mock/` dir | `make mockgen` | +| Both, or unsure | `make generate-all` | + +## 2. Run it + +```bash +$DEVC 'make proto-all' # or make mockgen / make generate-all +``` + +This is minutes-long; use a generous Bash timeout (600000). If the run fails +on protolock, the `.proto` change broke backward compatibility — read the +error; don't force it without flagging the compatibility break to the user. + +## 3. Verify before declaring success + +- `git status --porcelain` (on the host for a local devcontainer; inside via + `$DEVC 'git status --porcelain'` for a codespace) — the changed files should + be only the ones related to your proto/interface change plus their generated + outputs. **A diff touching every `.pb.go` in the repo means a wrong protoc + version — abort and check `$DEVC 'protoc --version'`.** +- Spot-check one regenerated file's header still says `protoc v4.23.4`: + `$DEVC 'grep -m1 "protoc " proto//.pb.go'` — run it through + `$DEVC`, not bare: in codespace mode the host clone is a *different* checkout, + so a bare `grep` would validate a stale file that was never regenerated. +- Build still compiles: `$DEVC 'make build-go'` (or the affected + `make build-`), and `$DEVC 'make gofmt'` after any Go changes. + +## Codespace caveat + +In a codespace the regenerated files land in the codespace's clone, not the +host repo. Commit/push from inside, or copy back with `gh codespace cp`. The +`status` subcommand of the devcontainer-run script tells you which mode you're in. diff --git a/.claude/skills/devcontainer-run/SKILL.md b/.claude/skills/devcontainer-run/SKILL.md new file mode 100644 index 0000000000..d693129bdb --- /dev/null +++ b/.claude/skills/devcontainer-run/SKILL.md @@ -0,0 +1,85 @@ +--- +name: devcontainer-run +description: >- + Detect the running Bucketeer dev container (local VS Code devcontainer or + GitHub Codespace) and run commands inside it. Use this whenever a task should + run in the dev container environment — make targets, builds, tests, kubectl / + helm / minikube commands, checking whether the container is up — or when the + user says "devcontainer-run", "devc", "dev container", "devcontainer", "codespace", or "run this + inside the container". Also use it when a task needs tools the container + guarantees but the host may lack (protoc 23.4, mockgen, protolock, helm, + kubectl, minikube). devcontainer-generate and devcontainer-deploy build on this skill. +--- + +# devcontainer-run — run commands inside the Bucketeer dev container + +The dev container is the canonical Bucketeer development environment: Ubuntu with +docker-in-docker, minikube + helm + kubectl, protoc v23.4, and Go tooling in +`/home/codespace/go-tools/bin` (a persistent volume, NOT on PATH in plain +non-login shells). The workspace is `/workspaces/bucketeer`, the user is +`codespace` (passwordless sudo). + +Human-facing documentation for this environment lives in `DEVELOPMENT.md` +("Working with the dev container from the host" and the Minikube sections); +this skill is the Claude-oriented operational version — when changing one, +keep the other in sync. + +## How to run anything inside it + +Always go through the wrapper script — it finds the container and sets up PATH: + +```bash +# Where is the container, and is the environment healthy? +bash .claude/skills/devcontainer-run/scripts/exec.sh status + +# Run any command in /workspaces/bucketeer inside the container +bash .claude/skills/devcontainer-run/scripts/exec.sh 'make build-api' +bash .claude/skills/devcontainer-run/scripts/exec.sh 'kubectl get pods' +``` + +Detection order (the script handles all of this): +1. Already inside the container (`/workspaces/bucketeer` exists, user `codespace`) → run directly. +2. Local devcontainer → `docker ps` filtered by label `devcontainer.local_folder=`, exec via `docker exec`. +3. GitHub Codespace → `gh codespace list` (needs the `codespace` auth scope), exec via `gh codespace ssh`. + +If more than one available Bucketeer codespace matches (a fork is also named +`bucketeer`, or you keep several), the script refuses to guess and exits 2 with +the list — pick one with `export BUCKETEER_CODESPACE=`. + +Exit code 2 means no container was found (or the codespace was ambiguous); the +script prints how to start one. +Don't fall back to running the command on the host in that case — tell the user +and let them choose, because host tool versions (especially protoc) may differ. + +## Local devcontainer vs Codespace — the one difference that matters + +- **Local devcontainer**: `/workspaces/bucketeer` is a bind mount of the host + repo. Files generated inside appear in the host working tree immediately. +- **Codespace**: a separate clone. Generated or edited files stay in the + codespace. To get them back: commit and push from inside, or + `gh codespace cp 'remote:/workspaces/bucketeer/' `. + Always tell the user which mode you're in when file changes are involved + (`status` prints it). + +## Environment facts and gotchas + +- Long commands (image builds, deploys) can take many minutes — use a generous + Bash timeout (600000) or `run_in_background`. +- `dockerd` inside the container is started by the post-attach hook, but that + only fires when an editor attaches. If `status` says it's not running: + `bash .claude/skills/devcontainer-run/scripts/exec.sh 'nohup sudo dockerd > /tmp/dockerd.log 2>&1 & sleep 5 && docker info > /dev/null && echo ok'` +- minikube must be started with `make start-minikube`, never `minikube start` + directly (the make target restores the cluster config and localenv services). + Note: `make start-minikube` intentionally **exits 1 if minikube is already + running** — check `minikube status` first instead of treating that as failure. +- `web-gateway.bucketeer.io` / `api-gateway.bucketeer.io` resolve via the + container's own `/etc/hosts` (pointed at `minikube ip`). Health checks with + curl against those hosts must run *inside* the container, not on the host. +- If go-tools are missing or permissions look broken, the fix is the setup + script: `bash .devcontainer/setup.sh` (idempotent, cache-aware). +- **Never run kubectl/helm bare on the host for dev work.** The host's kubectl + context may point at a real GKE cluster, not minikube — always go through the + wrapper so commands hit the cluster inside the container. +- The host may also run a docker-compose Bucketeer stack in parallel + (`docker-compose/compose.yml`). That is a different environment — this skill + is only about the dev container / minikube world. diff --git a/.claude/skills/devcontainer-run/scripts/exec.sh b/.claude/skills/devcontainer-run/scripts/exec.sh new file mode 100755 index 0000000000..af0521091b --- /dev/null +++ b/.claude/skills/devcontainer-run/scripts/exec.sh @@ -0,0 +1,150 @@ +#!/usr/bin/env bash +# Locate the Bucketeer dev container and run a command inside it. +# +# Usage: +# exec.sh status Report where the dev container is and its health +# exec.sh Run a command in /workspaces/bucketeer inside it +# +# Exit codes: 0 success, 2 no dev container found, otherwise the command's exit code. +set -euo pipefail + +WORKDIR=/workspaces/bucketeer +# go-tools live in a persistent volume; not on PATH in non-login shells +SETUP_PATH='export PATH=/home/codespace/go-tools/bin:$PATH' + +MODE="" +CID="" +CODESPACE="" + +detect() { + # Case 1: this shell is already inside the dev container + if [ -d "$WORKDIR" ] && [ "$(id -un)" = "codespace" ]; then + MODE=inside + return + fi + + # Case 2: local devcontainer (VS Code "Reopen in Container" / devcontainer CLI). + # Derive the repo root from this script's location (/.claude/skills/devcontainer-run/scripts/) + # so the exact label match works from any cwd — no fuzzy fallback that could pick + # the wrong container when multiple checkouts are running. + local repo_root + repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../../.." && pwd)" + CID="$(docker ps -q --filter "label=devcontainer.local_folder=$repo_root" 2>/dev/null | head -1 || true)" + if [ -n "$CID" ]; then + MODE=local + return + fi + + # Case 3: GitHub Codespace (requires `gh` with the codespace scope). + # Never silently pick one: a fork named `bucketeer`, or several codespaces on + # this repo, would otherwise run commands in the wrong clone/branch/cluster. + local candidates + candidates="$(gh codespace list --json name,repository,state \ + -q '.[] | select((.repository | endswith("/bucketeer")) and .state == "Available") | .name' 2>/dev/null \ + || true)" + if [ -n "${BUCKETEER_CODESPACE:-}" ]; then + if ! echo "$candidates" | grep -qx -- "$BUCKETEER_CODESPACE"; then + echo "BUCKETEER_CODESPACE='$BUCKETEER_CODESPACE' is not an available Bucketeer codespace." >&2 + echo "Available:" >&2 + echo "$candidates" | sed 's/^/ - /' >&2 + exit 2 + fi + CODESPACE="$BUCKETEER_CODESPACE" + MODE=codespace + return + fi + local count + count="$(echo "$candidates" | grep -c . || true)" + if [ "$count" -gt 1 ]; then + echo "Multiple available Bucketeer codespaces found — refusing to guess:" >&2 + echo "$candidates" | sed 's/^/ - /' >&2 + echo "" >&2 + echo "Pick one explicitly:" >&2 + echo " export BUCKETEER_CODESPACE=" >&2 + exit 2 + fi + if [ "$count" -eq 1 ]; then + CODESPACE="$candidates" + MODE=codespace + return + fi +} + +run_inside() { + local cmd="$1" + case "$MODE" in + inside) + bash -c "cd $WORKDIR && $SETUP_PATH && $cmd" + ;; + local) + docker exec -u codespace -w "$WORKDIR" "$CID" bash -c "$SETUP_PATH && $cmd" + ;; + codespace) + gh codespace ssh -c "$CODESPACE" -- "cd $WORKDIR && $SETUP_PATH && $cmd" + ;; + esac +} + +not_found() { + echo "No running Bucketeer dev container found." >&2 + echo "" >&2 + echo "Checked: this shell, local devcontainers (docker label devcontainer.local_folder)," >&2 + echo "and GitHub Codespaces (gh codespace list)." >&2 + echo "" >&2 + echo "To start one:" >&2 + echo " - VS Code: 'Dev Containers: Reopen in Container' on this repo" >&2 + echo " - CLI: devcontainer up --workspace-folder ." >&2 + echo " - Codespace: gh codespace create -R bucketeer-io/bucketeer" >&2 + if ! gh codespace list >/dev/null 2>&1; then + echo "" >&2 + echo "Note: 'gh codespace list' failed — if you use Codespaces, grant the scope with:" >&2 + echo " gh auth refresh -h github.com -s codespace" >&2 + fi + exit 2 +} + +status_report() { + case "$MODE" in + inside) echo "mode: inside (this shell is already in the dev container)" ;; + local) echo "mode: local devcontainer (docker exec, container $CID)" + echo "workspace: bind-mounted from the host — file changes appear in the host repo directly" ;; + codespace) echo "mode: GitHub Codespace '$CODESPACE' (gh codespace ssh)" + echo "workspace: SEPARATE clone — changes made inside do NOT appear in the host repo" ;; + esac + echo "---" + run_inside ' + echo "user: $(id -un) workdir: $(pwd)" + echo "protoc: $(protoc --version 2>/dev/null || echo MISSING)" + command -v mockgen >/dev/null && echo "go-tools: OK" || echo "go-tools: MISSING (run bash .devcontainer/setup.sh)" + docker info >/dev/null 2>&1 && echo "dockerd: running" || echo "dockerd: NOT running (start with: nohup sudo dockerd > /tmp/dockerd.log 2>&1 &)" + if minikube status >/dev/null 2>&1; then + # Always query the minikube context explicitly: the active context may point + # somewhere else entirely (a real GKE cluster), and reporting its pods here + # would be exactly the confusion this wrapper exists to prevent. + ctx=$(kubectl config current-context 2>/dev/null || echo unknown) + [ "$ctx" != "minikube" ] && echo "kube-context: WARNING active context is \"$ctx\", not minikube — bare kubectl/helm commands would hit that cluster; pass --context minikube / --kube-context minikube" + # Ignore transient states (Pending/ContainerCreating/Init) — batch CronJobs + # constantly spawn short-lived pods and would make the count flap. + if pods=$(kubectl --context minikube get pods --no-headers 2>/dev/null); then + total=$(echo "$pods" | grep -c . || true) + failing=$(echo "$pods" | grep -cE "CrashLoopBackOff|ImagePull|ErrImage|Error|OOMKilled|Evicted" || true) + echo "minikube: running ($total pods, $failing failing)" + [ "$failing" -gt 0 ] && echo "$pods" | grep -E "CrashLoopBackOff|ImagePull|ErrImage|Error|OOMKilled|Evicted" + else + echo "minikube: running, but kubectl failed to list pods in the minikube context — check kubectl config (kubectl config get-contexts)" + fi + else + echo "minikube: NOT running (start with: make start-minikube — never minikube start)" + fi + echo "git: $(git status --porcelain | wc -l | tr -d " ") modified files on branch $(git branch --show-current)" + ' +} + +detect +[ -z "$MODE" ] && not_found + +if [ "$#" -eq 0 ] || [ "$1" = "status" ]; then + status_report +else + run_inside "$*" +fi diff --git a/DEVELOPMENT.md b/DEVELOPMENT.md index b8171a115a..881b558cb3 100644 --- a/DEVELOPMENT.md +++ b/DEVELOPMENT.md @@ -24,6 +24,16 @@ container [here](https://docs.github.com/en/github/developing-online-with-codesp dev container) 4. Wait for the dev container to be ready +## Working with the dev container from the host + +If you edit code on the host (or drive the repo with tools running on the host), run environment-sensitive commands **inside the dev container**, not on the host: + +- **kubectl / helm**: the host's kubectl context may point at a completely different cluster (e.g. a real remote cluster) instead of the minikube instance inside the container. Always run cluster commands inside the container. +- **Code generation**: generated `.pb.go` files record the protoc version and must be produced with exactly protoc v23.4, which the dev container guarantees. A different host protoc rewrites the header of every generated file and makes the diff unreviewable. +- **Gateway health checks**: `web-gateway.bucketeer.io` / `api-gateway.bucketeer.io` are resolved by the *container's* `/etc/hosts` (pointing at `minikube ip`), so `curl` checks against those hosts only work inside the container. + +For Claude Code users, the project ships skills that automate this: `.claude/skills/devcontainer-run` detects the running dev container (local or Codespace) and runs commands inside it — `bash .claude/skills/devcontainer-run/scripts/exec.sh status` shows where it is and whether it's healthy. `devcontainer-generate` and `devcontainer-deploy` build on it for codegen and minikube deploys. + # Local Development Setup You can set up Bucketeer locally using one of two methods: @@ -57,6 +67,8 @@ make start-minikube **Note:** When you restart the Minikube cluster, you must use `make start-minikube` to start it. Do not use `minikube start` directly. +**Note:** `make start-minikube` intentionally exits with an error if minikube is already running. Check with `minikube status` first — an "already running" failure is not a problem to fix. + It will add 2 hosts to `/etc/hosts` that point to the minikube IP address: * `api-gateway.bucketeer.io` for API Gateway Service @@ -84,6 +96,15 @@ If you need to deploy a single service, you can do as follows. helm install backend manifests/bucketeer/charts/backend/ --values manifests/bucketeer/charts/backend/values.dev.yaml ``` +For faster iteration on a single service after a Go code change, rebuild and load the images, then restart only that deployment (`api`, `web`, `batch-server`, or `subscriber`): + +```shell +make build-go-embed +TAG=localenv make build-docker-images +TAG=localenv make minikube-load-images +kubectl rollout restart deployment web && kubectl rollout status deployment web +``` + **Note:** You can switch between data warehouses (MySQL, PostgreSQL, BigQuery) but remember to update the `values.dev.yaml` file to match the data warehouse you are using as the events persister and web service must use same event store service. **Note:** We use the `values.dev.yaml` file to override the default values in `values.yaml` file.