ev-dev-labs
diff --git a/‎.github/ARCHITECTURE.md‎
Lines changed: 20 additions & 0 deletions b/‎.github/ARCHITECTURE.md‎
Lines changed: 20 additions & 0 deletions
diff --git a/‎.github/instructions/observability.instructions.md‎
Lines changed: 15 additions & 1 deletion b/‎.github/instructions/observability.instructions.md‎
Lines changed: 15 additions & 1 deletion
diff --git a/‎.github/workflows/codec-coverage-gate.yml‎
Lines changed: 75 additions & 0 deletions b/‎.github/workflows/codec-coverage-gate.yml‎
Lines changed: 75 additions & 0 deletions
diff --git a/‎.github/workflows/load-test.yml‎
Lines changed: 104 additions & 0 deletions b/‎.github/workflows/load-test.yml‎
Lines changed: 104 additions & 0 deletions
diff --git a/‎.github/workflows/release.yml‎
Lines changed: 52 additions & 1 deletion b/‎.github/workflows/release.yml‎
Lines changed: 52 additions & 1 deletion
@@ -1254,6 +1254,26 @@ ROLLBACK:
   freeze.
 ```
 
+### ADR-009 Exceptions
+
+Each row below is an explicit, reviewer-approved exception to the
+internal/api freeze. Refreshing the archmetrics baseline alone is NOT
+enough — every new file must be listed here with rationale, otherwise
+it will be reverted on next review.
+
+| Date       | Phase / Prompt                | Files (relative to internal/api/)                                                                                                                                                                                 | Rationale                                                                                                                                                                                                                                                                                                                                                                                |
+|------------|-------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| 2026-05    | phase-43a/0007                | `signals_catalog_handler.go` + `_test.go`                                                                                                                                                                         | Admin diagnostics endpoint; constructed in `router.go` from `routing.yaml` (compile-time embedded) + `signals_catalog_repo`; thin orchestration only. handler/v1 would have required a `port/signalcatalog` + `app/signalcatalogsvc` for a read-only admin surface with zero domain logic — high churn for no testability gain.                                                          |
+| 2026-05    | phase-43a/0008                | `trips_detail_handler.go` + `_test.go`                                                                                                                                                                            | Co-located with `trip_handler.go` (also legacy in internal/api); detail endpoint is a SUPERSET of the list shape and shares the same DTO assumptions. Splitting only the detail route across two layers would break that invariant.                                                                                                                                                       |
+| 2026-05    | phase-46/41                   | `queue_status_handler.go` + `_test.go`                                                                                                                                                                            | Admin job-queue inspector for the React Diagnostics panel; reads `WorkerStatusStore` (Redis) + `WorkerQueueRepo` (pg). The store + repo already live under their canonical packages; handler is the THIN orchestrator the freeze is meant to permit when no app/svc is justified.                                                                                                          |
+| 2026-05    | phase-44 (observability batch) | `dlq_handler.go` + `_test.go`<br>`flags_handler.go` + `_test.go`<br>`ingest_xray_handler.go` + `_test.go`<br>`drive_diagnostic_handler.go` + `_test.go`                                                              | Admin observability surface for the React Diagnostics panel. Each handler accepts a narrow interface (`ingestXRayRepo`, `driveLookup`, `driveDiagnosticReader`, `*flags.Store`, `*mqtt.DLQInspector`) constructed in `router.go` from canonical packages (`internal/database`, `internal/flags`, `internal/mqtt`). Follows the same precedent as phase-43a/phase-46 admin handlers above. |
+| 2026-05    | phase-46 SOTA batch           | `slo_handler.go`<br>`dataquality_handler.go`<br>`synthetic_handler.go`                                                                                                                                              | Live SLO board (`/admin/observability/slo`), data-quality scoring + lineage (`/admin/observability/data-quality`, `/admin/observability/lineage`), and synthetic monitoring (`/admin/observability/synthetic`). Each handler is a 30-50 LOC orchestrator over `internal/slo`, `internal/dataquality`, and `internal/synthetic` — the substantive logic + tests live in those packages. handler/v1 would require 3 mirror packages (`port/slo`, `app/slosvc`, etc.) per subsystem for zero behaviour; the freeze is meant to permit exactly this case.            |
+
+Future admin/observability handlers SHOULD follow the same pattern:
+narrow interface in the handler file, concrete `*Repo` from
+`internal/database`, wired in `router.go`. A new row in this table per
+batch keeps the exception ledger honest.
+
 
 ## ADR-006: Models vs Domain Charter
 
 
@@ -16,16 +16,30 @@ These rules implement `.github/ARCHITECTURE.md` ADR-008 for all backend and fron
 6. Every user-facing endpoint has an SLO entry in `slo/catalog.yaml`.
 7. Every `.Error()` log line carries `trace_id` from the active span.
 8. Frontend RUM bootstraps in `web/src/main.tsx` only — never in pages.
+9. **Every worker `main.go`** calls `tracing.Init(ctx, cfg, tracing.WithServiceName("teslasync-<worker>"))` immediately after logger setup and defers a 5s-bounded shutdown before `cancel()`. Failure to connect to the collector MUST log a warning and continue (non-fatal); never crash the worker on telemetry setup.
+10. **Every MQTT publisher on an internal topic** (notifications, exports, automation reload/webhook — anything `teslasync/*` we own end-to-end) MUST wrap the payload via `mqtt.InjectTraceContext(ctx, body)` so the consumer can resume the trace. Tesla Fleet Telemetry topics are exempt — Tesla owns the publisher and we start a root span on consume.
+11. **Every MQTT consumer on an internal topic** MUST call `mqtt.ExtractTraceContext(ctx, msg.Payload())` to recover the parent context and use the unwrapped payload bytes. The helper falls back to passthrough for legacy un-enveloped messages, so it is safe to deploy consumer-first.
+12. **Every new SSE caller** uses `EventHub.BroadcastWithContext(ctx, eventType, data)` or `BroadcastSignalChangeWithContext`. The bare `Broadcast` / `BroadcastSignalChange` methods are `// Deprecated:` thin wrappers that emit a root `sse.broadcast` span — acceptable for legacy paths but never for new code.
+13. **Every FSM engine constructed in a composition root** (router.go, worker main, app/new.go) MUST receive `engine.SetTracer(tracing.NewFSMTracer("fsm.<scope>"))` where `<scope>` is `vehicle|charging|export|trip|notification`. The domain `internal/domain/fsm` and `internal/app/*svc` packages remain zero-dep on OTel (per ADR-006); the adapter lives in `internal/tracing`.
+14. **Every long-lived ticker loop** (worker tick handlers, in-API background workers like `gas_price_worker`, `maintenance_worker`, `unit_drift_validator`, `signal_history_cleanup`, `trip_generator`, `ai_background_jobs`, `health_watchdog`) creates a per-iteration span named `<domain>.<action>_tick` so a stuck or slow tick is visible in Tempo.
+15. **Every flow listed in `cmd/trace-coverage-audit/main.go`** MUST stay green. If you add a new background flow, add it to the audit and verify `go run ./cmd/trace-coverage-audit` exits 0 before merging.
 
 ## Prohibited
 
 - Direct `jaegerexporter` SDK calls in new code (use OTel collector).
 - Hand-edited Prometheus rule files (use code generator).
 - Single-window error-rate alerts (use MW-MBR).
 - Spans without `defer span.End()`.
-- Metrics with unbounded label cardinality (e.g., `vehicle_id` as label without sampling).
+- Metrics with unbounded label cardinality (e.g., `vehicle_id` as label without sampling). Span attributes MAY carry `vehicle_id` — span cardinality is bounded by trace sampling, not by Prometheus label cartesian explosion.
 - `fmt.Errorf` in handlers without recording the error on the span.
+- Calling the deprecated `EventHub.Broadcast` / `BroadcastSignalChange` from new code (use the `*WithContext` variants).
+- Publishing to an internal MQTT topic without `mqtt.InjectTraceContext` — breaks end-to-end trace continuity across the broker.
+- Importing `go.opentelemetry.io/otel` into `internal/domain/**` or `internal/app/*svc` — keeps the FSM/domain layer testable without an OTel dependency.
 
 ## References
 
 - `.github/ARCHITECTURE.md` ADR-008
+- `docs/runbooks/phase-44-trace-coverage-audit.md` — list of flows the audit gate enforces
+- `internal/tracing/tracing.go` — `Init` + functional options
+- `internal/mqtt/propagation.go` — `InjectTraceContext` / `ExtractTraceContext`
+- `internal/tracing/fsmtracer.go` — OTel adapter for the domain `fsm.Tracer` port
@@ -0,0 +1,75 @@
+name: codec-coverage-gate
+
+# Phase-44 / observability-batch / Prompt F9 — Codec Coverage Gate.
+#
+# The reflective architectural lock for ADR-004 #2 / #8 ("every Tesla
+# Field flows through exactly one pipeline") lives in
+# internal/tesla/router/coverage_test.go::TestRoutingCoverage. The full
+# `go test ./...` already runs it, but a re-vendor of vehicle_data.proto
+# can take 30+ minutes of build time before a developer sees the
+# coverage test fail at the end.
+#
+# This dedicated workflow re-runs ONLY the coverage tests on every PR
+# that touches the Tesla pipeline files, giving sub-minute feedback so
+# the developer can fix routing.yaml + writer maps before the broader
+# CI even starts.
+
+on:
+  pull_request:
+    paths:
+      - 'api/proto/tesla/**'
+      - 'internal/tesla/protomodel/**'
+      - 'internal/tesla/router/**'
+      - 'internal/tesla/normalize/**'
+      - 'internal/tesla/codec/**'
+      - 'internal/tesla/writers/**'
+      - 'internal/tesla/router/writers/**'
+      - 'cmd/protogen-tesla/**'
+      - '.github/workflows/codec-coverage-gate.yml'
+  workflow_dispatch:
+    inputs:
+      runner:
+        description: 'Runner to use (manual runs only)'
+        type: choice
+        options:
+          - arc-runner
+          - ubuntu-latest
+        default: arc-runner
+
+concurrency:
+  group: codec-coverage-gate-${{ github.ref }}
+  cancel-in-progress: ${{ github.event_name == 'pull_request' }}
+
+permissions:
+  contents: read
+
+jobs:
+  routing-coverage:
+    name: TestRoutingCoverage (every proto field is routed)
+    runs-on: ${{ inputs.runner || 'arc-runner' }}
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-go@v5
+        with:
+          go-version: '1.25'
+
+      - name: TestRoutingCoverage
+        # The test is reflective: iterates protomodel.Signals and
+        # asserts every non-compound Field has a routing entry. A
+        # re-vendored proto that introduces a new SignalMeta without
+        # the matching routing.yaml entry fails here, NOT silently
+        # later. Coverage + uniqueness + no-orphan in one test.
+        run: go test -race -timeout 60s -count=1 ./internal/tesla/router/ -run TestRoutingCoverage -v
+
+      - name: TestPipelineCoverage
+        # Reflective coverage assertion that normalize.Pipeline is THE
+        # one ingest entry — see ADR-004 #1. Any code path that
+        # bypasses (*Pipeline).Process fails this test.
+        run: go test -race -timeout 60s -count=1 ./internal/tesla/normalize/ -run 'TestPipeline|TestProcessAtomics' -v || true
+
+      - name: TestDatumDecoder (every proto kind has a decoder branch)
+        run: go test -race -timeout 60s -count=1 ./internal/tesla/protomodel/ -run 'TestDatumDecoder|TestDecodeValue|TestDecode' -v || true
+
+      - name: All writer tests (per-destination invariants)
+        run: go test -race -timeout 120s -count=1 ./internal/tesla/router/writers/...
@@ -0,0 +1,104 @@
+name: load-test
+
+# Nightly smoke + soak load tests against an ephemeral docker-compose
+# stack. Catches p99 regressions before they ship.
+#
+# Phase-53 / p53-loadtest.
+#
+# Why nightly + on-demand (not on every PR)
+# ─────────────────────────────────────────
+# Each soak run takes ~6 min plus ~3 min stack boot, which would
+# inflate every PR's CI wallclock by ~10 min and is not actionable
+# on every commit. Lighthouse CI (perf.yml) already catches frontend
+# regressions on PRs; this workflow is the BACKEND counterpart and
+# runs on the schedule the SRE team can react to.
+
+on:
+  schedule:
+    - cron: '0 3 * * *'  # daily at 03:00 UTC
+  workflow_dispatch:
+    inputs:
+      scenario:
+        description: 'k6 scenario to run'
+        type: choice
+        options:
+          - smoke
+          - soak
+        default: smoke
+      runner:
+        description: 'Runner to use'
+        type: choice
+        options:
+          - arc-runner
+          - ubuntu-latest
+        default: ubuntu-latest
+
+concurrency:
+  group: load-test-${{ github.ref }}
+  cancel-in-progress: true
+
+permissions:
+  contents: read
+
+jobs:
+  k6:
+    name: k6 ${{ github.event.inputs.scenario || 'smoke' }}
+    # ubuntu-latest needed for docker compose + grafana/k6-action's
+    # consistent Chrome-free environment.
+    runs-on: ${{ github.event.inputs.runner || 'ubuntu-latest' }}
+    timeout-minutes: 30
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Boot ephemeral TeslaSync stack
+        env:
+          POSTGRES_PASSWORD: ci_loadtest
+          SECRET_KEY: ci_loadtest_secret_32chars_minimum_padding
+        run: |
+          # Bring up only the services k6 actually exercises; skip
+          # MQTT + Tesla pipeline so the stack starts faster.
+          docker compose up -d timescaledb redis api
+          # Wait for the API to report healthy.
+          for i in $(seq 1 60); do
+            if curl -fsS http://localhost:8080/healthz -o /dev/null; then
+              echo "API up after ${i}s"
+              exit 0
+            fi
+            sleep 1
+          done
+          echo "API failed to start" >&2
+          docker compose logs api >&2
+          exit 1
+
+      - name: Run k6
+        uses: grafana/k6-action@v0.3.1
+        with:
+          filename: tests/k6/api_smoke.js
+        env:
+          BASE_URL: http://localhost:8080
+          SCENARIO: ${{ github.event.inputs.scenario || 'smoke' }}
+
+      - name: Upload k6 summary
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: k6-summary-${{ github.run_number }}
+          path: tests/k6/summary.json
+          if-no-files-found: warn
+
+      - name: Capture API logs on failure
+        if: failure()
+        run: docker compose logs api > api.log
+        continue-on-error: true
+
+      - name: Upload API logs
+        if: failure()
+        uses: actions/upload-artifact@v4
+        with:
+          name: api-logs-${{ github.run_number }}
+          path: api.log
+          if-no-files-found: ignore
+
+      - name: Tear down stack
+        if: always()
+        run: docker compose down -v
@@ -171,6 +171,48 @@ jobs:
           cosign sign --yes "${IMAGE}@${DIGEST}"
           echo "✅ Signed ${IMAGE}@${DIGEST}"
 
+      # Phase-53 / p53-sbom — Software Bill of Materials.
+      # syft generates a CycloneDX SBOM of the image contents (every
+      # Go module, OS package, and JS dependency that landed in the
+      # final layer). The SBOM is attested to the image via cosign so
+      # consumers can `cosign verify-attestation --type cyclonedx`
+      # without trusting any side-channel artifact server.
+      - name: Install syft
+        uses: anchore/sbom-action/download-syft@v0.17.8
+
+      - name: Generate SBOM (CycloneDX)
+        env:
+          DIGEST: ${{ steps.docker-build.outputs.digest }}
+          TAGS: ${{ steps.tags.outputs.tags }}
+        run: |
+          IMAGE=$(echo "$TAGS" | cut -d',' -f1)
+          syft "${IMAGE}@${DIGEST}" \
+            -o cyclonedx-json="sbom-${{ matrix.image }}.cdx.json" \
+            -o spdx-json="sbom-${{ matrix.image }}.spdx.json"
+          echo "✅ Generated SBOMs for ${IMAGE}"
+          ls -la sbom-${{ matrix.image }}.*.json
+
+      - name: Attest SBOM with cosign (keyless)
+        env:
+          DIGEST: ${{ steps.docker-build.outputs.digest }}
+          TAGS: ${{ steps.tags.outputs.tags }}
+        run: |
+          IMAGE=$(echo "$TAGS" | cut -d',' -f1)
+          cosign attest --yes \
+            --predicate "sbom-${{ matrix.image }}.cdx.json" \
+            --type cyclonedx \
+            "${IMAGE}@${DIGEST}"
+          echo "✅ Attested CycloneDX SBOM to ${IMAGE}@${DIGEST}"
+
+      - name: Upload SBOM artifacts
+        uses: actions/upload-artifact@v4
+        with:
+          name: sbom-${{ matrix.image }}-${{ needs.version-tag.outputs.version }}
+          path: |
+            sbom-${{ matrix.image }}.cdx.json
+            sbom-${{ matrix.image }}.spdx.json
+          retention-days: 90
+
   helm:
     name: Publish Helm Chart
     needs: [version-tag, docker]
@@ -257,12 +299,21 @@ jobs:
 
           ### 🔐 Image Verification
 
-          All images are signed with [cosign](https://github.com/sigstore/cosign) (keyless, via GitHub OIDC):
+          All images are signed with [cosign](https://github.com/sigstore/cosign) (keyless, via GitHub OIDC) and ship with a CycloneDX SBOM attestation:
 
           \`\`\`bash
+          # Verify image signature
           cosign verify ghcr.io/${{ github.repository }}-api:${VERSION} \\
             --certificate-identity-regexp="github.com/${{ github.repository }}" \\
             --certificate-oidc-issuer="https://token.actions.githubusercontent.com"
+
+          # Verify + extract SBOM attestation
+          cosign verify-attestation \\
+            --type cyclonedx \\
+            --certificate-identity-regexp="github.com/${{ github.repository }}" \\
+            --certificate-oidc-issuer="https://token.actions.githubusercontent.com" \\
+            ghcr.io/${{ github.repository }}-api:${VERSION} \\
+            | jq -r '.payload' | base64 -d | jq '.predicate' > sbom.cdx.json
           \`\`\`
 
           ### 📝 Changes