Merge branch 'main' into dependabot/pip/src/agentflow_runtime/processing/flink_jobs/pydantic-gte-2.13.5-and-lt-3

Author: brownjuly2003-code

.github/CODEOWNERS

@@ -0,0 +1,13 @@
+# Repository-side ownership for review-sensitive surfaces.
+/.github/** @brownjuly2003-code
+/infrastructure/** @brownjuly2003-code
+/helm/** @brownjuly2003-code
+/src/agentflow_runtime/serving/api/auth/** @brownjuly2003-code
+/src/agentflow_runtime/serving/api/middleware/** @brownjuly2003-code
+/scripts/release.py @brownjuly2003-code
+/scripts/evaluate_trivy_policy.py @brownjuly2003-code
+/scripts/run_pip_audit_scan.py @brownjuly2003-code
+/security/trivy-waivers.json @brownjuly2003-code
+/config/project_claims.toml @brownjuly2003-code
+/SECURITY.md @brownjuly2003-code
+/docs/release-*.md @brownjuly2003-code

.github/workflows/ci.yml

@@ -300,27 +300,57 @@ jobs:
         run: |
           # Security-critical auth manager (key match/verify, tenant isolation,
           # rate-limit/failed-auth windows, rotation grace) and a mutmut target;
-          # the gate runs its dedicated unit files. Module coverage is 94% so the
+          # the gate runs its dedicated unit files. Module coverage is 97% so the
           # 90% gate keeps a cushion; the remaining gap is the platform-divergent
-          # SIGHUP handler and bcrypt rotation paths the integration/e2e auth
-          # suites cover.
+          # SIGHUP handler, the two read-only-store branches load() reaches only
+          # when the store flips between probe and write, and the thin
+          # KeyRotator/usage-table delegations the integration/e2e auth suites
+          # cover.
+          #
+          # This list IS the gate: a dedicated unit file that is not named here
+          # buys the module nothing. test_key_store_readonly.py arrived with the
+          # read-only Secret mount work (F-02 B, 2026-08-23) and was never added,
+          # which by itself took the module from 94% to 82% -- unnoticed, because
+          # an earlier step in this job was failing and this one never ran.
           #
           # NOTE: unlike the other per-module gates this uses `coverage run` +
           # `coverage report --include`, NOT `pytest --cov=<module>`. The auth
           # manager pulls in duckdb (usage table), and pytest-cov's source
           # instrumentation of a duckdb-importing module trips duckdb's lazy
           # `_duckdb._sqltypes` import at COLLECTION time, both locally and on CI
           # runners. `coverage run` imports duckdb normally and avoids the break.
-          python -m coverage run -m pytest tests/unit/test_auth.py tests/unit/test_auth_manager_pure_logic.py tests/unit/test_auth_manager_memory_bounds.py tests/unit/test_auth_hashed_key_guidance.py tests/unit/test_auth_argon2_lookup.py -p no:schemathesis
+          python -m coverage run -m pytest \
+            tests/unit/test_auth.py \
+            tests/unit/test_auth_manager_pure_logic.py \
+            tests/unit/test_auth_manager_memory_bounds.py \
+            tests/unit/test_auth_hashed_key_guidance.py \
+            tests/unit/test_auth_argon2_lookup.py \
+            tests/unit/test_auth_key_store_probe.py \
+            tests/unit/test_auth_manager_key_resolution.py \
+            tests/unit/test_key_store_readonly.py \
+            -p no:schemathesis
           python -m coverage report --include="*/serving/api/auth/manager.py" --show-missing --fail-under=90
       - name: Run key rotation coverage gate
         run: |
           # Security-critical key-rotation lifecycle (create/rotate/revoke,
           # grace-period scheduling, rotation status) and a mutmut target. Like
           # the auth manager gate it pulls in duckdb, so it uses coverage run +
           # coverage report --include (not pytest --cov) to avoid the
-          # duckdb _duckdb._sqltypes collection break. Module coverage is 93%.
-          python -m coverage run -m pytest tests/unit/test_key_rotation.py -p no:schemathesis
+          # duckdb _duckdb._sqltypes collection break.
+          #
+          # Module coverage is 100%. It was 89.6% on test_key_rotation.py alone
+          # -- passing only because coverage rounds to 90 before comparing, so
+          # one more defensive line would have turned this gate red without
+          # anyone changing its behaviour. test_key_rotation.py pins the happy
+          # lifecycle; the revoke_and_failures file adds what it never reached:
+          # revoke_key_by_id (the only revoke path the admin router calls since
+          # F-02 A), the read-only vs merely-broken key-store split, and the
+          # grace-period cleanup failure that runs on a timer thread with no
+          # caller left to see it.
+          python -m coverage run -m pytest \
+            tests/unit/test_key_rotation.py \
+            tests/unit/test_key_rotation_revoke_and_failures.py \
+            -p no:schemathesis
           python -m coverage report --include="*/serving/api/auth/key_rotation.py" --show-missing --fail-under=90
       - name: Run outbox coverage gate
         run: |
@@ -460,6 +490,10 @@ jobs:
       - name: Install dependencies (frozen lock profile)
         run: bash scripts/ci_sync.sh dev-tools
       - uses: azure/setup-helm@9bc31f4ebc9c6b171d7bfbaa5d006ae7abdb4310 # v5.0.1
+        with:
+          version: v3.16.3
+      - name: Report Helm version
+        run: helm version --short
       - uses: helm/kind-action@ef37e7f390d99f746eb8b610417061a60e82a6cc # v1.14.0
         with:
           install_only: true
@@ -541,12 +575,12 @@ jobs:
 
   terraform-validate:
     runs-on: ubuntu-latest
-    timeout-minutes: 10
+    timeout-minutes: 20
     steps:
       - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
       - uses: hashicorp/setup-terraform@dfe3c3f87815947d99a8997f908cb6525fc44e9e # v4.0.1
         with:
-          terraform_version: "1.8.0"
+          terraform_version: "1.15.4"
       - name: Terraform fmt check
         run: terraform fmt -check -recursive infrastructure/terraform/
       - name: Terraform init
@@ -557,6 +591,18 @@ jobs:
         run: |
           cd infrastructure/terraform
           terraform validate
+      # h1: hashes in the lock file carry no platform label. This step is the
+      # guard that linux_amd64 (ubuntu-latest) is actually covered; it uses the
+      # same three-platform command the module README documents (re-run
+      # 2026-09-05: no lock rewrite). Three provider zips (~0.7 GB) plus
+      # terraform init is why this job's timeout-minutes is 20, not 10.
+      # It has not been observed on a GitHub runner until the owner pushes.
+      - name: Provider lock covers linux_amd64
+        run: |
+          cd infrastructure/terraform
+          terraform providers lock -platform=linux_amd64 -platform=darwin_arm64 -platform=windows_amd64
+          git ls-files --error-unmatch .terraform.lock.hcl
+          git diff --exit-code .terraform.lock.hcl
 
   sdk-ts:
     # Audit P1-5: typecheck/tests/build for the TypeScript SDK previously ran

.github/workflows/security.yml

@@ -47,6 +47,7 @@ jobs:
         run: |
           python - <<'PY'
           import os
+          import shutil
           import subprocess
           import sys
           import tempfile
@@ -84,6 +85,22 @@ jobs:
               deduped = list(dict.fromkeys(entries))
               target.write_text("\n".join(deduped) + "\n", encoding="utf-8")
 
+          def copy_locked_requirements(source: Path, target: Path) -> int:
+              if not source.exists():
+                  raise SystemExit(f"Required dependency lock is missing: {source}")
+              locked = [
+                  line.strip()
+                  for line in source.read_text(encoding="utf-8").splitlines()
+                  if line and not line[0].isspace() and not line.startswith("#")
+              ]
+              if not locked:
+                  raise SystemExit(f"Required dependency lock resolved to zero packages: {source}")
+              if any("==" not in requirement or not requirement.endswith("\\") for requirement in locked):
+                  raise SystemExit(f"Required dependency lock is not fully hash-pinned: {source}")
+              shutil.copyfile(source, target)
+              print(f"Copied {len(locked)} locked packages -> {target}")
+              return len(locked)
+
           def resolve_requirements(name: str, entries: list[str], target: Path) -> int:
               if not entries:
                   raise SystemExit(
@@ -172,13 +189,13 @@ jobs:
               )
 
           # The Flink runtime is not an extra (its beam chain can never
-          # co-install with core pyarrow>=17) — its manifest lives next to
-          # the Flink image Dockerfile. Scan it as its own bucket.
-          flink_count = resolve_requirements(
-              "flink-runtime",
-              load_requirements(
-                  root / "src" / "agentflow_runtime" / "processing" / "flink_jobs" / "requirements.txt"
-              ),
+          # co-install with core pyarrow>=17). Keep validating its human-owned
+          # manifest, but scan the image's hash lock without resolving it again.
+          load_requirements(
+              root / "src" / "agentflow_runtime" / "processing" / "flink_jobs" / "requirements.txt"
+          )
+          flink_count = copy_locked_requirements(
+              root / "src/agentflow_runtime/processing/flink_jobs/flink-requirements.lock",
               output_dir / "requirements-flink-runtime.txt",
           )
 
@@ -254,14 +271,39 @@ jobs:
           # `from apache_beam.io.parquetio import *`, and parquetio's class
           # bodies evaluate `pa.Table` annotations with pa=None ->
           # AttributeError). The dependency is load-bearing at import time
-          # even though our jobs never use it.
+          # even though our jobs never use it. Expires 2026-10-27 (same as
+          # the Trivy waiver). Disposition is recorded in
+          # security/trivy-waivers.json (flink-runtime, CVE-2026-25087).
+          #
+          # SFTY-20260724-05622 (CVE-2026-59939): unbounded decompression of
+          # HTTP response bodies encoded with Content-Encoding: gzip or
+          # deflate in _decompressContent (httplib2/__init__.py), fixed in
+          # 0.32.0. A malicious or compromised HTTP server can return a
+          # small compressed payload that expands to an arbitrarily large
+          # size in memory (MemoryError / OOM-kill). Only the flink-runtime
+          # bucket resolves an affected httplib2 (apache-flink's beam chain
+          # caps httplib2<0.23). Why the risk is negligible here:
+          #   1. The Flink DataStream job has no httplib2 import or HTTP
+          #      client path. Its ingress is Kafka, and it never processes
+          #      attacker-controlled compressed HTTP response bodies.
+          # Blocked upstream, not by our pin: apache-flink==2.3.0 requires
+          # apache-beam>=2.54.0,<=2.61.0; apache-beam==2.61.0 requires
+          # httplib2>=0.8,<0.23.0. Installing httplib2 0.32 into this graph
+          # is impossible without breaking Beam. Dependabot watches the
+          # Flink manifest (src/agentflow_runtime/processing/flink_jobs),
+          # so the next apache-flink release opens a PR — re-check this
+          # ignore there. Remove this ignore when a supported apache-flink/
+          # apache-beam dependency set accepts httplib2>=0.32.0, or
+          # immediately if an httplib2 call path is added. Expires 2026-10-27
+          # (same as the Trivy waiver).
+          # Disposition is recorded in security/trivy-waivers.json
+          # (flink-runtime, CVE-2026-59939).
           #
           # 88512 (langchain cross-ecosystem false positive) was ignored here
           # until 2026-07-20; PyUp corrected the entry (verified: safety
           # 2.3.5 reports 0 findings on the pinned langchain stack), so the
           # ignore is gone. If it ever resurfaces, it fails this job loudly.
-          safety check \
-            --ignore SFTY-20260217-93940 \
+          python scripts/run_safety_scan.py --waivers security/trivy-waivers.json \
             -r .artifacts/security/safety/requirements-main.txt \
             -r .artifacts/security/safety/requirements-sdk.txt \
             -r .artifacts/security/safety/requirements-integrations.txt \
@@ -287,6 +329,9 @@ jobs:
           python-version: "3.11"
       - name: Install pip-audit
         run: python -m pip install --upgrade pip "pip-audit>=2.7,<3" uv==0.8.23
+      # Deliberately bare: the production image lock is the one inventory with
+      # no waiver mechanism reachable at all. A finding here is a release
+      # blocker, not something to argue about in security/trivy-waivers.json.
       - name: Audit the locked production dependency set
         run: pip-audit --no-deps -r requirements-docker.lock
       # Audit F-03: the production lock alone left uv.lock's dev/test chain
@@ -302,7 +347,17 @@ jobs:
           mkdir -p .artifacts/security/pip-audit
           uv export --frozen --format requirements-txt --all-extras \
             --no-emit-project -o .artifacts/security/pip-audit/requirements-all-profiles.txt
-          pip-audit --no-deps -r .artifacts/security/pip-audit/requirements-all-profiles.txt
+          # Audit FB-02: nltk 3.10.3 carries PYSEC-2026-3740 and upstream has
+          # published no fix, so no bump can close it and this job stayed red
+          # with nowhere to record why. The suppression lives in
+          # security/trivy-waivers.json (scope python-profiles), where it is
+          # validated, expires on 2026-11-01, and fails this job the moment it
+          # stops matching -- including the day nltk ships a fix. Do not pass
+          # --ignore-vuln here: a flag in YAML has none of those properties.
+          python scripts/run_pip_audit_scan.py \
+            --waivers security/trivy-waivers.json \
+            --scope python-profiles \
+            -r .artifacts/security/pip-audit/requirements-all-profiles.txt
 
   npm-audit:
     runs-on: ubuntu-latest

.github/workflows/terraform-apply.yml

@@ -32,6 +32,8 @@ jobs:
       - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
 
       - uses: hashicorp/setup-terraform@dfe3c3f87815947d99a8997f908cb6525fc44e9e # v4.0.1
+        with:
+          terraform_version: "1.15.4"
 
       - name: Resolve Terraform tfvars file
         id: tfvars
@@ -84,6 +86,8 @@ jobs:
       - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
 
       - uses: hashicorp/setup-terraform@dfe3c3f87815947d99a8997f908cb6525fc44e9e # v4.0.1
+        with:
+          terraform_version: "1.15.4"
 
       - name: Resolve Terraform tfvars file
         id: tfvars
@@ -138,6 +142,8 @@ jobs:
       - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
 
       - uses: hashicorp/setup-terraform@dfe3c3f87815947d99a8997f908cb6525fc44e9e # v4.0.1
+        with:
+          terraform_version: "1.15.4"
 
       - name: Resolve Terraform tfvars file
         id: tfvars

.gitignore

@@ -35,7 +35,6 @@ logs/
 .terraform/
 *.tfstate
 *.tfstate.backup
-.terraform.lock.hcl
 
 data/
 checkpoints/
@@ -135,10 +134,12 @@ agentflow_demo_api.duckdb*
 # Claude Code config/session state and any *_local/ scratch dir are kitchen, never
 # product — untracked-but-unignored is one `git add -A` from a public repo. (N-6/P5.2)
 /.claude/
+/.myflow/
 /*_local/
 /AGENT_STATE.md
 /AUTOPILOT.md
 /BACKLOG.md
+/SESSION_LOG*.md
 /help.md
 /res.md
 /all-open-questions-closure-plan.md

AGENTS.md

@@ -0,0 +1,15 @@
+# DE_project Rules
+
+## Durable State and Autonomous Continuation
+- On a new session, resume, compaction, stale context, or contradiction, rebuild from `git status` and these files when present: `AGENT_STATE.md`, `docs/SESSION_HANDOFF.md`, `docs/operations/local-verification-matrix.md`, `AUTOPILOT.md`, `docs/operations/autonomous-compact-safe-process.md`, `BACKLOG.md`, `.autopilot/BLOCKED.md`.
+- If remembered work is already finished, impossible, or contradicted by disk state, stop pursuing it and choose another safe atomic item.
+- Candidate order is dirty WIP, failed or blocked verification diagnostics, PLAN/Next Work, narrow failing or uncovered tests, then documented operations findings.
+- A real external blocker ends that atomic item. Record non-secret evidence and continue only with an independent safe item.
+- A single verification or acceptance check may have at most **five full attempts per named slice**, counted `1/5` through `5/5`; the initial run is `1/5`. After each failed or timed-out attempt, another full attempt is allowed only after a new narrowed diagnostic hypothesis or a relevant corrective change. Raw unchanged retries are forbidden. If attempt `5/5` fails or times out, end the slice for the turn and report it.
+
+## Verification Hosts
+- Keep local Windows Python processes below 1 GiB. Use narrow checks and rely on `D:\SystemState\PythonMemoryGuard\python-memory-guard.ps1`.
+- Run Docker-heavy verification, jobs likely to exceed 1 GiB, and macOS diagnostics through SSH alias `deproject-mac`; the Mac checkout is `/Users/julia/agentflow-docker-check`.
+- The sharded Windows suite pins `OPENBLAS_NUM_THREADS=1` and `OMP_NUM_THREADS=1` in every child process; unpinned, OpenBLAS alone accounts for ~550 MiB of the 1 GiB budget. Measurements and the production caveat: `docs/operations/windows-verification.md`.
+- Non-secret Mac context belongs in `AGENT_STATE.md`, `docs/SESSION_HANDOFF.md`, and `docs/operations/local-verification-matrix.md`.
+- Never print, commit, or copy SSH keys, passwords, tokens, recovery codes, or other secrets into project files, logs, PRs, or chat.