fix(ci): stabilize optional dependency resolution

Author: JuliaEdom

docs/PROJECT_CLOSURE.md

@@ -58,8 +58,8 @@ rehearsal and third-party penetration test.
 ## Обязательные внешние closure gates
 
 - closing commits `59fc19c` (Operator/Kafka reproducibility) and `72b9609`
-  (Python compatibility/core lock remediation) published to `main`; the final
-  core-only/Ruff/runtime-image hardening commit remains to be published;
+  (Python compatibility/core lock remediation), plus `f11fd59`
+  (core-only/Ruff/runtime-image hardening), are published to `main`;
 - all required GitHub checks green on the exact closing SHA;
 - two open Dependabot PRs receive a final one-by-one disposition;
 - the substantial post-v2 changes receive a final release/version decision and
@@ -74,9 +74,19 @@ Push, PR mutations and release actions require explicit owner authorization.
   core-only API import without `pyiceberg`) are fixed and locally verified.
 - The Security Scan findings were traced to packages vendored by runtime
   `pip`; the final image now removes installer tooling after `pip check`.
+- CI on `f11fd59` passed every separate workflow and every early main-CI gate,
+  then exposed two upstream-resolution boundaries: PyIceberg 0.11 writes
+  without native core and MCP 2.0 breaking the 1.x integration API. The
+  candidate now resolves `pyiceberg-core==0.7.0` across Python 3.11–3.13 and
+  constrains MCP to `<2`; `2170` local unit/property tests and all `39`
+  selected clean-Mac dependency tests pass. See
+  [dependency-compatibility-2026-07-30.md](dependency-compatibility-2026-07-30.md).
 - Independent Mac image validation reports zero HIGH/CRITICAL findings with
   Trivy 0.70.0; see
   [security-runtime-image-trivy-2026-07-30.md](security-runtime-image-trivy-2026-07-30.md).
+- Per owner allocation, Codex performed the dependency remediation,
+  documentation, Mac verification, and Git closeout; Grok/Grokw remained
+  paused to preserve its remaining weekly quota.
 - Production status remains `candidate`; none of this closes the live Iceberg,
   restore/replay, fresh soak/rollback, external pen-test, or npm approval gates.
 

docs/STATUS.md

@@ -26,10 +26,15 @@ Closing CI hardening on 2026-07-30 also verified the Python 3.11/3.12/3.13
 compatibility lanes, made `pyiceberg` optional for the core-only API path,
 aligned SDK examples with Ruff 0.16 Markdown formatting, and removed
 `pip`/`setuptools`/`wheel` from the final API image after `pip check`.
+A later dependency-resolution gate now keeps the Iceberg write path complete
+with Python 3.13-compatible `pyiceberg-core==0.7.0` and holds the optional MCP
+integration on its supported 1.x API.
 A clean core-only HTTP smoke returned 200 for health, entity, and NL query.
 An independent Mac rebuild scanned with Trivy 0.70.0 at zero HIGH/CRITICAL
 findings; see
 [security-runtime-image-trivy-2026-07-30.md](security-runtime-image-trivy-2026-07-30.md).
+The dependency failure, resolution, and clean Mac verification are recorded in
+[dependency-compatibility-2026-07-30.md](dependency-compatibility-2026-07-30.md).
 The exact post-remediation GitHub SHA must still complete all required checks
 before the external closure gate is claimed.
 
@@ -53,6 +58,7 @@ recorded in [PROJECT_CLOSURE.md](PROJECT_CLOSURE.md).
 | Clean-checkout PyFlink OCI build + submission smoke | PASS clean-checkout OCI build + submission smoke on 2026-07-30 — image built, JobID `RUNNING`, not Operator/E2E | [perf/golden-flink-submission-2026-07-30.md](perf/golden-flink-submission-2026-07-30.md) |
 | Flink Kubernetes Operator + Helm golden deploy | PASS clean kind Operator/Helm deploy of verified image on exact HEAD `36ed1ec` — CR/job stable, checkpoints `2→23`, leader flaps `0`, not lake E2E | [perf/golden-operator-acceptance-2026-07-30.md](perf/golden-operator-acceptance-2026-07-30.md) |
 | Hardened API runtime image (local acceptance) | core-only API import/HTTP smoke PASS; Trivy 0.70.0 reports 0 HIGH/CRITICAL after runtime installer removal | [security-runtime-image-trivy-2026-07-30.md](security-runtime-image-trivy-2026-07-30.md) |
+| Python cloud/MCP dependency compatibility (local + isolated Mac) | 2170 unit/property tests PASS; MCP 1.29 + PyIceberg 0.11.1/core 0.7.0 clean environment and 39 focused tests PASS | [dependency-compatibility-2026-07-30.md](dependency-compatibility-2026-07-30.md) |
 
 ## 2026-07-23 audit closure
 

docs/dependency-compatibility-2026-07-30.md

@@ -0,0 +1,91 @@
+# Dependency compatibility remediation — 2026-07-30
+
+## Scope
+
+This record covers the Python cloud/Iceberg and optional MCP dependency
+boundaries discovered by the final CI run. It does not claim production
+acceptance or replace the remaining live golden-topology gates.
+
+After the owner asked to preserve the remaining Grok/Grokw weekly quota, Codex
+performed all diagnosis, remediation, Mac verification, documentation, and Git
+closeout work described here. Grok was not invoked for this remediation.
+
+## Trigger
+
+GitHub Actions CI run
+[`30573486366`](https://github.com/brownjuly2003-code/agentflow/actions/runs/30573486366)
+tested commit `f11fd59fea61fc7bf29d3fa8f8c41139c236cfb4`. Its lint,
+lock, SDK, Terraform, schema, Helm, and Python 3.11/3.12/3.13 compatibility
+gates passed. Separate Contract, E2E, Flink Smoke, Load, Security Scan,
+Scorecard, and Staging Deploy workflows also passed.
+
+Two dependency boundaries still failed:
+
+- the integration job resolved `pyiceberg==0.11.1` without
+  `pyiceberg-core`; five Iceberg write/materialization tests raised
+  `NotInstalledError`;
+- `integrations[mcp]` allowed an unbounded major upgrade, so the unit job
+  resolved `mcp==2.0.0`; that release removed the MCP 1.x `Tool.inputSchema`
+  model field and low-level `Server.list_tools()` decorator used by the
+  integration.
+
+## Remediation
+
+The cloud extra now installs:
+
+```toml
+"pyiceberg>=0.7,<1",
+"pyiceberg-core>=0.7,<0.8",
+```
+
+PyIceberg 0.11 uses the native package for partition transforms and table
+appends. The previously selected core 0.8 distribution excludes Python 3.13;
+core 0.7 provides a stable-ABI wheel across the supported Python 3.11–3.13
+matrix. `uv.lock` and the hash-locked Docker export both resolve
+`pyiceberg-core==0.7.0`.
+
+The MCP extra now declares `mcp>=1.0,<2`. This preserves the integration's
+documented MCP 1.x API instead of silently accepting a breaking major release.
+A unit dependency contract pins both boundaries so future configuration or
+lock drift fails before the runtime jobs.
+
+## Verification
+
+- RED contract stage: two new dependency contracts failed against the old
+  declarations.
+- GREEN targeted stage: `38 passed` locally across dependency contracts, MCP,
+  and non-Docker Iceberg scenarios.
+- Full Windows Python 3.13 unit/property suite: `2170 passed, 1 warning`.
+- `uv 0.8.23`: `uv lock --check` passed; the generated lock/export diff adds
+  only `pyiceberg-core==0.7.0`.
+- Isolated Mac clone:
+  `/tmp/agentflow-ci-f11-codex-deps-01`, based on exact `f11fd59` plus the
+  five candidate dependency files.
+- Clean Mac Python 3.13 environment:
+  `mcp==1.29.0`, `pyiceberg==0.11.1`,
+  `pyiceberg-core==0.7.0`, and `pip check` passed.
+- Mac contract/MCP/Iceberg selection: all `39` tests passed, including the
+  Docker REST-catalog materialization case.
+- Direct OSV queries for the three resolved packages returned no vulnerability
+  records.
+
+The changed Docker lock was also rebuilt through `Dockerfile.api` on Mac:
+
+```text
+agentflow-api:security-codex-f11deps-01
+manifest list sha256:95254620bf98e73ce89feae28e0b178a3cf00956c7e63eedd40f8958705645d1
+```
+
+The build installed `pyiceberg-core==0.7.0`, completed `pip check`, and then
+removed `pip`, `setuptools`, and `wheel`. Runtime API import returned
+`AgentFlow Query API`; `python -m pip` remained absent. Trivy `0.70.0` with
+`HIGH,CRITICAL`, `--ignore-unfixed`, and `--exit-code 1` exited `0`; every
+reported OS and Python package row was clean.
+
+## Evidence boundary
+
+This is local and isolated-Mac acceptance of the dependency candidate. The
+exact pushed source state must still complete the repository's GitHub CI and
+Security Scan. Published-image signing, external penetration testing, live
+lake-to-serving materialization, restore/replay, and soak/rollback remain
+separate gates.

docs/release-readiness.md

@@ -87,9 +87,16 @@ The 2026-07-30 closing remediation additionally covers two clean-install
 boundaries: the core-only API imports and serves health/entity/query without
 the optional `pyiceberg` package, and the final API image removes
 `pip`/`setuptools`/`wheel` after its hash-locked install and `pip check`.
+The cloud profile separately includes `pyiceberg-core==0.7.0`, whose
+stable-ABI build supports the full Python matrix and restores PyIceberg 0.11
+write transforms; the optional MCP integration is capped below the breaking
+2.0 API. Clean Mac Python 3.13 verification passed with MCP 1.29 and all 39
+selected contract/MCP/Iceberg tests.
 Independent Mac validation with Trivy 0.70.0 reported zero HIGH/CRITICAL
 findings; details and evidence limits are in
-[security-runtime-image-trivy-2026-07-30.md](security-runtime-image-trivy-2026-07-30.md).
+[security-runtime-image-trivy-2026-07-30.md](security-runtime-image-trivy-2026-07-30.md)
+and
+[dependency-compatibility-2026-07-30.md](dependency-compatibility-2026-07-30.md).
 All required GitHub checks must still be green on the exact final pushed SHA.
 
 ## Scope

docs/security-audit.md

@@ -162,6 +162,13 @@ reported zero HIGH/CRITICAL findings while the API import remained healthy.
 See
 [security-runtime-image-trivy-2026-07-30.md](security-runtime-image-trivy-2026-07-30.md).
 
+The same closeout also converted two implicit dependency assumptions into
+tested supply-chain boundaries: MCP is constrained to its supported 1.x major
+API, and PyIceberg's write-time native core is explicitly locked to the
+Python 3.11–3.13-compatible 0.7 line. The regenerated hash lock, clean Mac
+environment, rebuilt image, OSV queries, and Trivy result are recorded in
+[dependency-compatibility-2026-07-30.md](dependency-compatibility-2026-07-30.md).
+
 The Bandit baseline currently records a historical `B310` finding in `src/serving/backends/clickhouse_backend.py`. SQL construction findings are not globally suppressed; reviewed identifier construction is handled through narrow suppressions and tests.
 
 Helm defaults no longer embed production-shaped API-key verifier hashes. Operators can render a chart-managed Secret for local use or mount an existing Kubernetes Secret, which is friendlier to External Secrets Operator, Sealed Secrets, or equivalent workflows.

docs/security-runtime-image-trivy-2026-07-30.md

@@ -66,6 +66,26 @@ Runtime checks:
 - Trivy summary: Debian `13.5` vulnerabilities `0`; every detected Python
   package vulnerabilities `0`.
 
+## Dependency-lock revalidation
+
+Final CI dependency diagnosis added `pyiceberg-core==0.7.0` to the cloud
+profile and hash lock. Codex rebuilt the image from isolated Mac clone
+`/tmp/agentflow-ci-f11-codex-deps-01`, based on exact `f11fd59` plus the five
+candidate dependency files:
+
+```text
+agentflow-api:security-codex-f11deps-01
+manifest list sha256:95254620bf98e73ce89feae28e0b178a3cf00956c7e63eedd40f8958705645d1
+```
+
+The rebuilt Python 3.11 image installed core 0.7.0 through
+`requirements-docker.lock`, passed `pip check`, removed installer tooling,
+and preserved the API import. A fresh official Trivy `0.70.0` container scan
+with the same HIGH/CRITICAL failure policy exited `0`; every reported OS and
+Python package row was clean. Dependency root-cause and Python 3.13/MCP
+verification are in
+[dependency-compatibility-2026-07-30.md](dependency-compatibility-2026-07-30.md).
+
 ## Evidence boundary
 
 This is content-level local acceptance of the candidate runtime image. The

integrations/pyproject.toml

@@ -17,7 +17,8 @@ dependencies = [
 
 [project.optional-dependencies]
 mcp = [
-    "mcp>=1.0",
+    # MCP 2 removes the low-level Server decorators used by this integration.
+    "mcp>=1.0,<2",
 ]
 
 [project.scripts]

pyproject.toml

@@ -40,6 +40,9 @@ dependencies = [
 cloud = [
     "boto3>=1.35,<2",
     "pyiceberg>=0.7,<1",
+    # PyIceberg 0.11 write paths require native transforms. Core 0.8 excludes
+    # Python 3.13, while 0.7 provides a stable-ABI wheel across Python 3.11-3.13.
+    "pyiceberg-core>=0.7,<0.8",
 ]
 postgres = [
     # ADR 0010 slice 5: PostgresControlPlaneStore (scale profile). Optional

requirements-docker.lock

@@ -1148,6 +1148,14 @@ pyiceberg==0.11.1 ; python_full_version < '3.14' \
     --hash=sha256:e273242cdca56029af694d7ce18075d47a74d034326d663ff6dd2655a6f44825 \
     --hash=sha256:eb3a0a3e630ee89758eb96b39b456f4697732351fb0c080e9498ea578f9b71f9
     # via agentflow-runtime
+pyiceberg-core==0.7.0 ; python_full_version < '3.14' \
+    --hash=sha256:2303276b0d8b57b8ce4a3ef9e5050ffdefd9561684728cd0a9a72a9ad0a6a74d \
+    --hash=sha256:64dd3c3c7af6d39097e1d070161e98118e5dd423392992a9074e1ab813fd467b \
+    --hash=sha256:74543885cc97e8d976707f97dd01e8f03dce8a5d3a01e41ada0aaefa13c742f9 \
+    --hash=sha256:8166883ace30a388d2f659634bec87731cad7bc52341c997fcdd4e13780e4345 \
+    --hash=sha256:94fe0281f09c84cdd4e56d217865ef1d81e0cb0b708ba6eb2f4c5ae8bf86f0ba \
+    --hash=sha256:99e23463c30c4180329719fe1f120e779b20616a36bbdd42042b70063a13bd39
+    # via agentflow-runtime
 pyparsing==3.3.2 ; python_full_version < '3.14' \
     --hash=sha256:850ba148bd908d7e2411587e247a1e4f0327839c40e2e5e6d05a007ecc69911d \
     --hash=sha256:c777f4d763f140633dcb6d8a3eda953bf7a214dc4eff598413c070bcdc117cbc

tests/unit/test_contract_dependencies.py

@@ -85,11 +85,12 @@ def test_runtime_and_docker_image_include_redis_client():
     assert "redis==" in docker_lock
 
 
-def test_cloud_extra_uses_pure_python_pyiceberg_without_native_core():
-    """Cloud must ship pure-Python pyiceberg; pyiceberg-core has no cp313 wheels.
+def test_cloud_extra_uses_python_313_compatible_pyiceberg_core():
+    """Iceberg writes need native transforms without reintroducing the cp313 break.
 
-    The native extra is an optimization unused by project code. Pinning it via
-    ``pyiceberg[pyiceberg-core]`` blocks Python 3.13 lock/install on CI.
+    PyIceberg 0.11 requires ``pyiceberg-core`` for partition transforms and
+    table appends. Core 0.8 excludes Python 3.13, while 0.7 ships a compatible
+    stable-ABI wheel for every supported interpreter.
     """
     pyproject = _load_pyproject()
     cloud = pyproject["project"]["optional-dependencies"]["cloud"]
@@ -100,9 +101,18 @@ def test_cloud_extra_uses_pure_python_pyiceberg_without_native_core():
         for dependency in cloud
     ), cloud
     assert not any("[pyiceberg-core]" in dependency for dependency in cloud), cloud
-    assert not any(dependency.startswith("pyiceberg-core") for dependency in cloud), cloud
+    assert "pyiceberg-core>=0.7,<0.8" in cloud
     assert "pyiceberg==" in docker_lock
-    assert "pyiceberg-core==" not in docker_lock
+    assert "pyiceberg-core==0.7.0" in docker_lock
+
+
+def test_mcp_integration_rejects_breaking_major_versions():
+    integrations = tomllib.loads(
+        (PROJECT_ROOT / "integrations" / "pyproject.toml").read_text(encoding="utf-8")
+    )
+    mcp_dependencies = integrations["project"]["optional-dependencies"]["mcp"]
+
+    assert "mcp>=1.0,<2" in mcp_dependencies
 
 
 def test_contract_workflow_uses_contract_extra():