Skip to content

Commit 8d65495

Browse files
JuliaEdomclaude
andcommitted
fix(flink): stop shipping pip's vendored packages in the job image
The Flink image's last two unwaived HIGH findings were msgpack 1.1.2 (GHSA-6v7p-g79w-8964) and setuptools 70.3.0 (CVE-2025-47273). Neither is installed: they are pip's own vendored copies, and Trivy reads pip/_vendor/vendor.txt as an installed package list. The versions match pip 26.2.1's vendor set exactly, and neither package has a dist-info under /opt/pyflink-venv, which is what identified the source. Neither could be fixed where it was. No pin changes what pip vendors, so a bump is not available; and both advisories have upstream fixes, so neither qualifies for a waiver -- security/trivy-waivers.json waives what upstream has not fixed (FB-02), and the flink scope's two existing waivers (pyarrow, httplib2) are exactly that shape. Nothing installs at runtime, so the build removes pip once the venv is complete. The job's dependencies are the hash-locked flink-requirements.lock set, and PyFlink shells out to pip only when python.requirements is configured, which no job here sets. The API image has shipped without pip since P1-3. Two properties make the removal safe rather than hopeful: it runs in the RUN that installed the requirements, so no earlier layer keeps pip's files for an image pulled at that layer; an `import apache_beam, pyflink` immediately after it fails the build if the venv no longer starts, instead of surfacing in flink-smoke or in production. tests/unit/test_flink_image_pip_removal.py pins both properties plus the advisory ids in the rationale, so the next edit that reaches for "just bump pip" has to confront why that does not work. All five cases go red against the pre-change Dockerfile. Verified locally: ruff lint/format clean, the new file and the existing test_flink_image_policy.py pass (15 passed), docs ratchets pass (57 passed), mkdocs build --strict clean. The image rebuild and the Trivy verdict are CI's trivy job; the job still submitting is CI's flink-smoke. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018ybRVCaL1Lyd9d5kFTndh3
1 parent 32955f7 commit 8d65495

4 files changed

Lines changed: 122 additions & 0 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,30 @@ All notable changes to AgentFlow are documented in this file.
44

55
## [Unreleased]
66

7+
### Security — the Flink image stops shipping pip's vendored packages
8+
9+
* **pip's vendored dependency set was the image's last two unwaived HIGH
10+
findings.** Trivy reads `pip/_vendor/vendor.txt` as installed packages, so
11+
msgpack 1.1.2 (GHSA-6v7p-g79w-8964) and setuptools 70.3.0 (CVE-2025-47273)
12+
were reported against `agentflow-flink` even though no code imports either.
13+
Neither is fixable in place: no pin changes what pip vendors, and both
14+
advisories are fixed upstream, so neither qualifies for a waiver — the rule
15+
`security/trivy-waivers.json` enforces is *upstream has published no fix*
16+
(FB-02).
17+
* **Nothing installs at runtime, so the build removes pip** once the venv is
18+
complete; the API image has shipped without it since P1-3. The job's
19+
dependencies are the hash-locked `flink-requirements.lock` set, and PyFlink
20+
shells out to pip only when `python.requirements` is configured, which no
21+
job here sets.
22+
* **The removal is proven, not assumed.** It runs in the same layer that
23+
installed the requirements — a later `RUN` would leave pip's files in an
24+
earlier layer, where an image pulled at that layer still carries them — and
25+
an `apache_beam`/`pyflink` import immediately after it fails the *build* if
26+
the venv no longer starts, instead of failing the smoke job or production.
27+
* **A shell in the container now has no `pip install`.**
28+
`docs/operations/flink-operators.md` says so, and says what to do instead:
29+
change the lock and rebuild.
30+
731
### Quality — the auth-manager coverage gate was measuring 82%, not 94%
832

933
* **The file list *is* the gate.** `Run auth manager coverage gate` runs the

‎docs/operations/flink-operators.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -120,6 +120,13 @@ standard `/docker-entrypoint.sh`. PyFlink and job dependencies live in
120120
`/opt/pyflink-venv`; both rendered jobs pass that interpreter through
121121
`-pyclientexec`.
122122

123+
That venv ships no `pip`. pip carries vendored copies of its own
124+
dependencies (`pip/_vendor`), and those copies were the image's last two
125+
unwaived HIGH findings -- unfixable in place, because no pin changes what pip
126+
vendors. Nothing installs at runtime, so the build removes pip once the venv
127+
is complete. A shell in the container therefore has no `pip install`: change
128+
`flink-requirements.lock` and rebuild the image instead.
129+
123130
## Kubernetes Operator 1.15 compatibility
124131

125132
Install the pinned Apache Flink Kubernetes Operator 1.15.0 chart before the

‎src/agentflow_runtime/processing/flink_jobs/Dockerfile‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,17 @@ ARG CURL_GUARDS="--connect-timeout 30 --speed-limit 65536 --speed-time 60 --retr
2121
USER root
2222
# BuildKit cache reuses a verified JAR across CI's same-runner build retry.
2323
# Cache hits skip the network; corrupt/partial entries are discarded.
24+
#
25+
# pip is removed once the venv is built. pip vendors its own dependency set
26+
# (pip/_vendor/vendor.txt), which Trivy reports as installed packages: msgpack
27+
# 1.1.2 (GHSA-6v7p-g79w-8964) and setuptools 70.3.0 (CVE-2025-47273) were the
28+
# image's last two unwaived HIGH findings, and both are unfixable in place --
29+
# no pin can change what pip vendors, and both advisories are fixed upstream,
30+
# so neither qualifies for a waiver under security/trivy-waivers.json. Nothing
31+
# installs at runtime: the job's dependencies are the hash-locked set above,
32+
# and PyFlink only shells out to pip when python.requirements is configured,
33+
# which no job here sets. The API image drops pip for the same reason. The
34+
# import check after the removal proves the venv still runs the job's stack.
2435
RUN --mount=type=cache,id=flink-sql-connector-kafka-5.0.0-2.2,target=/var/cache/flink-connectors \
2536
apt-get update \
2637
&& apt-get install -y --no-install-recommends ca-certificates curl python3 python3-pip python3-venv \
@@ -63,6 +74,8 @@ RUN --mount=type=cache,id=flink-sql-connector-kafka-5.0.0-2.2,target=/var/cache/
6374
&& /opt/pyflink-venv/bin/python -m pip install --no-cache-dir --upgrade pip \
6475
&& /opt/pyflink-venv/bin/python -m pip install --no-cache-dir --require-hashes -r /tmp/flink-requirements.lock \
6576
&& /opt/pyflink-venv/bin/python -c "from importlib.metadata import version; v = version('apache-flink'); assert v == '${FLINK_VERSION}', f'flink-requirements.lock installed apache-flink {v}, ARG FLINK_VERSION is ${FLINK_VERSION}'" \
77+
&& /opt/pyflink-venv/bin/python -m pip uninstall --yes pip \
78+
&& /opt/pyflink-venv/bin/python -c "import apache_beam, pyflink; from pyflink.datastream import StreamExecutionEnvironment" \
6679
&& rm -rf /var/lib/apt/lists/* /tmp/flink-requirements.lock
6780

6881
WORKDIR /opt/agentflow
Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,78 @@
1+
"""Contract: the Flink job image does not ship pip.
2+
3+
pip vendors its own dependency set (``pip/_vendor/vendor.txt``) and Trivy
4+
reports those vendored copies as installed packages. msgpack 1.1.2
5+
(GHSA-6v7p-g79w-8964) and setuptools 70.3.0 (CVE-2025-47273) were the image's
6+
last two unwaived HIGH findings and are unfixable in place: no pin changes what
7+
pip vendors, and both advisories have upstream fixes, so neither is waivable
8+
under ``security/trivy-waivers.json``. Removing pip once the venv is built is
9+
the fix, and it is only safe while nothing installs at runtime.
10+
11+
Docker is not available on the Windows development host, so these checks read
12+
the Dockerfile text. The image rebuild plus Trivy proof is the CI ``trivy`` job;
13+
the job actually starting is the CI ``flink-smoke`` job.
14+
"""
15+
16+
from __future__ import annotations
17+
18+
from pathlib import Path
19+
20+
PROJECT_ROOT = Path(__file__).resolve().parents[2]
21+
DOCKERFILE = PROJECT_ROOT / "src" / "agentflow_runtime" / "processing" / "flink_jobs" / "Dockerfile"
22+
23+
VENV_PYTHON = "/opt/pyflink-venv/bin/python"
24+
REQUIREMENTS_INSTALL = f"{VENV_PYTHON} -m pip install --no-cache-dir --require-hashes"
25+
PIP_REMOVAL = f"{VENV_PYTHON} -m pip uninstall --yes pip"
26+
IMPORT_CHECK = f'{VENV_PYTHON} -c "import apache_beam, pyflink;'
27+
28+
29+
def _dockerfile() -> str:
30+
return DOCKERFILE.read_text(encoding="utf-8")
31+
32+
33+
def test_the_runtime_venv_does_not_keep_pip() -> None:
34+
text = _dockerfile()
35+
36+
assert text.count(PIP_REMOVAL) == 1
37+
38+
39+
def test_pip_is_removed_after_the_locked_requirements_are_installed() -> None:
40+
text = _dockerfile()
41+
42+
install_at = text.index(REQUIREMENTS_INSTALL)
43+
removal_at = text.index(PIP_REMOVAL)
44+
45+
assert install_at < removal_at
46+
47+
48+
def test_pip_is_removed_in_the_same_layer_that_installed_it() -> None:
49+
text = _dockerfile()
50+
51+
# A removal in a later RUN would leave pip's files in the earlier layer,
52+
# and an image pulled at that layer still carries the vendored packages.
53+
venv_run = next(block for block in text.split("\nRUN ") if REQUIREMENTS_INSTALL in block)
54+
55+
assert PIP_REMOVAL in venv_run
56+
57+
58+
def test_the_venv_is_proven_to_still_import_the_job_stack_after_the_removal() -> None:
59+
text = _dockerfile()
60+
61+
removal_at = text.index(PIP_REMOVAL)
62+
import_at = text.index(IMPORT_CHECK)
63+
64+
# Without this the build would happily produce an image whose Python
65+
# environment no longer starts, and the failure would land in flink-smoke
66+
# (or production) instead of the build.
67+
assert removal_at < import_at
68+
assert "StreamExecutionEnvironment" in text[import_at : import_at + 200]
69+
70+
71+
def test_the_reason_pip_cannot_simply_be_upgraded_is_recorded() -> None:
72+
text = _dockerfile()
73+
74+
# The next person to reach for "just bump pip" needs the two advisory ids
75+
# and the reason a waiver is not available for them.
76+
assert "GHSA-6v7p-g79w-8964" in text
77+
assert "CVE-2025-47273" in text
78+
assert "pip/_vendor" in text

0 commit comments

Comments
 (0)