Skip to content

fix(openhands): ship 0.71.3 with security-patched images - #1387

Merged
dylan-openhands merged 22 commits into
release/0.71from
dj/c3-0713-chart
Oct 14, 2026
Merged

dylan-openhands merged 22 commits into
release/0.71from
dj/c3-0713-chart

Conversation

@dylan-openhands

@dylan-openhands dylan-openhands commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

What

This is 0.71.3 for the 0.71.x hotfix line, which ships to C3. It starts from 0.71.1's charts/ and replicated/, so none of 0.71.2's Copa-patched images carry forward. Each image is then re-pinned to a rebuilt image or a newer image from the same release line. .github/, including the Beta deploy guard, and the chart version are kept from release/0.71; release-please proposes 0.71.3.

Tracking: PLTF-3723.

Where each image comes from

Image 0.71.3 source
enterprise-server 1.64.0-r1 OpenHands/enterprise#734. The tag retags the PR build that was tested and scanned, so the digest is the candidate's (6c137216…)
runtime-api 0.10.0-r1 OpenHands/runtime-api#771, built from tag v0.10.0-r1
agent-server 1.49.5-r1-python OpenHands/software-agent-sdk#5691, built from tag 1.49.5-r1
agent-canvas 1.23.0-r1 OpenHands/OpenHands#18258, built from tag v1.23.0-r1 on agent-server 1.49.5-r1-python, with the prod PostHog key
Keycloak 26.8.0, PostgreSQL 16.4, Redis 7.4.3, Laminar Postgres 15.19 Same-version rebuilds from OpenHands/third-party-images. Keycloak 26.8.0 is the cherry-pick of #1399
Laminar frontend-ee and app-server-ee lmnr 0.1.825, pinned by digest; previously the floating latest
Laminar chart 0.2.5, curl 8.22.0, ClickHouse 26.3-alpine, Valkey 9.1.2-alpine Backports of #1364, #1362, #1366
LiteLLM 1.100.5, kubectl v1.33.13, cert-manager 1.20.4, trust-pkg, alpine/openssl 3.5.9 Patch bumps on the same release line
Postgres client used by init containers and jobs One digest-pinned global.postgresClientImage (alpine/psql:18.6), replacing 11 hardcoded bitnamilegacy/postgresql:latest / postgres:14 references

Vulnerability result

These are the images C3's Helm values enable, the same set as their own scan. Both versions were rendered and scanned with Trivy on the 2026-10-12 DB, HIGH/CRITICAL, unique CVE × package. The four OpenHands images were scanned on linux/amd64 and linux/arm64, with identical results.

Fixable C / H Total C / H
0.71.1 102 / 897 119 / 1710
0.71.3 (this PR) 3 / 15 17 / 724

All 3 / 15 remaining fixable findings are in agent-server, and agent-canvas reports the same 18. None is exploitable in the image:

  • 17 are VS Code built-in extension manifests (extensions/{handlebars,npm,grunt,json,ini,diff}/package.json) that Trivy misreads as the npm libraries of the same name.
  • 1 is setuptools CVE-2025-47273, matched on pip's vendored pkg_resources, which doesn't contain the vulnerable package_index.py.

The gh, Docker Buildx, Docker Compose and RootlessKit findings are fixed: #5691 rebuilds the first three from upstream source with patched Go modules and takes RootlessKit 3.3.0. An OpenVEX document for the 18 is in security/agent-server-1.49.5-hotfix.openvex.json in software-agent-sdk, scoped to the candidate digests.

The per-image table is on PLTF-3723. Images that a default KOTS install runs but C3 doesn't (MinIO, sysbox, alpine/k8s and others) weren't rescanned for this version.

Fleet upgrade test (shared-2)

shared-2 was clean-installed on 0.71.1 (Replicated sequence 1863), then upgraded through the admin console to this PR at 505c0f1 (sequence 1864, 0.71.3-rc.1), on the bundled Postgres. That run used the candidate builds of the four OpenHands images. The release builds differ from them only in the agent-server Go tool rebuilds and RootlessKit 3.3.0, plus the prod PostHog key in agent-canvas.

  • Rollout: every pod came up Running. LiteLLM restarted once while Postgres was restarting, then stayed up. A sandbox created before the upgrade keeps its old image until it restarts, as expected.
  • Keycloak migration: Keycloak migrated both realms from 26.3.0 through to 26.8.0 and started in 16s, with 0 ERROR log lines. The realm-provisioning init container re-applied the realm cleanly.
  • Keycloak data: before the upgrade, a probe user was seeded with a GitHub link, an online session and an offline session.
    • After the upgrade, both refresh tokens issued by 26.3 still refresh.
    • userinfo and introspection work with the new tokens, and a fresh password login works.
    • The link, attributes and offline session survived.
    • Identity providers (5, with stored tokens on) and clients (7) are unchanged.
    • A real GitHub-signed-in account kept its link and attributes.

Notes

  • The alpine/psql commit (ce65a45) is the version tested and scanned above. Dropping it puts the postgres client back on postgres:16.15-alpine, which has 1 / 24 fixable, all in the unused gosu.
  • Keep ghcr.io/openhands/patched/agent-server@sha256:c143… published. Sandboxes created on 0.71.2 keep using that image when resumed after the upgrade.

Upgrade notes for C3

  • Keycloak's database migrates one way from 26.3 to 26.8 on first start. Back it up first; the chart README's "Keycloak 26.3 to 26.8" section has the commands, and helm upgrade prints a warning. Rolling back needs that backup.
  • Laminar's ClickHouse data upgrades in place from 25.10 to 26.3 on first start, with no rollback.

Testing

  • helm lint passes, helm unittest passes 169/169, and pytest gives 211 passed and 4 skipped locally. The 5 collection errors are modules missing from the local environment; CI's test-scripts passes.
  • helm template with C3's values resolves enterprise-server, runtime-api and agent-server to the release digests above.
  • Earlier, per-component checks are unchanged:
    • Every postgres-client call site was exercised against Bitnami 16.4, postgres:13 and postgres:18 over TLS.
    • PostgreSQL and Redis were smoke-tested in containers with data-dir and rollback checks.
    • LiteLLM 1.100.1 → 1.100.5 has no pending migrations.

🤖 Generated with Claude Code

Base for 0.71.3: 0.71.2's Copa image pins don't carry forward. charts/ and
replicated/ are restored from openhands/0.71.1; .github/ and the chart
version (0.71.2) are kept so release-please cuts 0.71.3.
Follows the restore: 0.71.2 pointed this test at the Copa-patched paths.
Rebuilt hotfix releases (1.64.0-r1, 1.49.5-r1-python) name the release they rebuild; release_tag() now strips -rN as it already strips -patched and digest pins.
Pins the PR-build candidate digests of the 7 in-house images, default ghcr.io/openhands/* repositories. The -r1 tags do not exist yet; the tag text is ignored because a digest is present. Replace each digest with the rescanned -r1 release digest once the hotfix PRs are tagged.

agent-canvas carries the STAGING PostHog key (OpenHands#18258 candidate build) and must never ship: it needs the manual cancel-and-dispatch release before this pin is final.

agent-server tag keeps 1.49.5 as its first segment so enterprise's get_agent_server_image() rewrite (default repository only) leaves it unchanged.
… and jobs

Every wait-for-db, create-db and create-db-user container hardcoded either bitnamilegacy/postgresql:latest or postgres:14. All but Keycloak's bypassed the Replicated proxy and the airgap registry rewrite, and :latest was unpinned.

Route all of them through global.postgresClientImage, pinned by digest to postgres:16.15-alpine, and mirror it in the online and local-registry Replicated values. Keycloak's wait-for-db keeps uid 1001, which the Bitnami image used to supply.
Optional on top of the previous commit: drop it to stay on the Docker Official postgres:16.15-alpine image.

postgres:16.15-alpine still reports 1 CRITICAL and 24 HIGH fixable findings, all in the Go stdlib of its gosu binary, which none of these containers run. alpine/psql 18.6 ships only the client tools and scans clean (0 CRITICAL/HIGH). psql 18 supports servers back to 9.2, and the image runs every site's command with output identical to the old images against PostgreSQL 13, 16 and 18.
@github-actions

Copy link
Copy Markdown
Contributor

⚠️ Chart appVersion has drifted from the image tag

The openhands chart's appVersion no longer matches the deployed image tag. These are meant to stay in sync.

Location Value
charts/openhands/Chart.yaml → appVersion 1.64.0
charts/openhands/values.yaml → image.tag 1.64.0-r1@sha256:6c137216f1c6afe1d560300e92990a52050064a1358472c314285ae8e0bf8307

Fix: set appVersion to 1.64.0-r1@sha256:6c137216f1c6afe1d560300e92990a52050064a1358472c314285ae8e0bf8307 in charts/openhands/Chart.yaml.

This is a notice, not a blocking check.

@github-actions github-actions Bot added the type: fix A bug fix label Oct 10, 2026
Changing either chart makes release-please propose a release of it on
release/0.71. The sync check now reads 1.49.5-r1-python as 1.49.5-python,
so image-loader can stay on the release tag, as it did for 0.71.2. The
crd-check edit was a doc comment (the chart is consumed as OCI 0.1.0).
dylan-openhands and others added 8 commits October 12, 2026 18:23
…on third-party-images rebuilds

Same upstream versions rebuilt in OpenHands/third-party-images with current
Debian packages and Go helpers. Fixable C/H (2026-10-12 DB): keycloak
23/155 -> 6/59, postgresql 17/67 -> 0/0, redis 17/74 -> 0/0, Laminar
postgres 1/24 -> 0/0. Airgap repository names are unchanged because the
images keep their last path segment.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Both ran on the floating latest tag. 0.1.825 (lmnr build of 2026-10-12, the current latest) has no fixable HIGH/CRITICAL findings.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Cherry-picked from main (26eaaa4). Conflict in values.yaml resolved to the 26.8.0 tag plus its digest.

(cherry picked from commit 26eaaa4)
… test builds

agent-server 973f316 (software-agent-sdk#5691, Debian snapshot 20261012T234422Z) and the agent-canvas build on top of it (OpenHands#18258). Both are 3/20 fixable; the released -r1 digests replace these before 0.71.3 is cut.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@dylan-openhands dylan-openhands changed the title fix(openhands): rebuild 0.71.3 images without Copa fix(openhands): ship 0.71.3 with security-patched images Oct 14, 2026
@aivong-openhands

Copy link
Copy Markdown
Contributor

Validation: Helm upgrade 0.71.1 → this PR (505c0f1) on a local kind cluster

Setup: single-node kind (Kubernetes 1.26). Chart 0.71.1 installed from GHCR (not the Replicated registry), Traefik ingress, bundled Postgres, Redis, and Keycloak, Laminar enabled. The values mirror a Helm self-hosted install: replicated.enabled: false, Keycloak extraEnvVars set as a custom list (so the chart's KC_HTTP_ENABLED default is replaced), LiteLLM pinned to v1.92.0. The upgrade used this PR's chart with two image overrides: agent-server 8bf9c1c-python@sha256:9e2aac… (software-agent-sdk#5691) and LiteLLM 1.100.5@sha256:65cdef…. Login is GitHub through Keycloak.

How the test exercises the change: the upgrade swaps every bundled image (Keycloak bitnamilegacy 26.3.0 → third-party-images 26.8.0, bitnamilegacy Postgres and Redis → third-party-images, the -r1 app images, Laminar 0.1.825), so it covers the one-way Keycloak schema migration and a restart of the bundled Postgres under existing data.

Check 0.71.1 baseline After upgrade
Keycloak schema (migration_model) 26.3.0 26.8.0; log shows each step 26.4.0 → 26.8.0, 0 ERROR lines
Keycloak realm data (realms, users, 5 IdPs, GitHub-linked user, allhands client) recorded unchanged
Keycloak starts with the custom extraEnvVars list (no KC_HTTP_ENABLED) — yes
App data (conversations, users, org LLM profile) 2 / 1 / set unchanged
Running images bitnamilegacy Keycloak/Postgres/Redis, LiteLLM v1.92.0, agent-server 1.49.5 third-party-images Keycloak/Postgres/Redis, LiteLLM 1.100.5, sandbox on agent-server digest 9e2aac…
Fresh GitHub login after upgrade — works
New conversation, agent replies to a test prompt pass pass (cold sandbox, ~60s to reply)
Laminar traces from the sandbox — appear in the Laminar project

Upgrade result: the first helm upgrade was marked failed, and a re-run of the same command succeeded. The post-upgrade hook budget-reconcile-gate started while openhands-postgresql-0 was restarting for its new image (the image pull took 3m36s on kind). The hook got connection refused from openhands-postgresql:5432 and exited non-zero. The job has backoffLimit: 0 and no wait-for-database step, so Helm records the release as failed even though every resource rolled out. Any upgrade that restarts the bundled Postgres can hit this when Postgres is not back before the hook connects. Re-running the same helm upgrade with Postgres up gave revision 3 deployed.

Sandbox image label: the sandbox pod spec shows agent-server:1.49.5-python@sha256:9e2aac…. The digest is the 8bf9c1c image and the running imageID confirms it, but the tag in the name is the old default.


This comment was drafted by an AI agent on behalf of the user.

@aivong-openhands

aivong-openhands commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Validation: Helm upgrade 0.71.1 → this PR (505c0f1) on GKE

Setup: GKE 1.36.4-gke.1391002, e2-standard-8 nodes, Traefik, Let's Encrypt TLS. Chart 0.71.1 with bundled Postgres, Redis and Keycloak, Laminar enabled, and GitHub login through Keycloak. The values are the same style as the kind run: Keycloak extraEnvVars as a custom list, LiteLLM pinned to v1.92.0. Like the kind run, the upgrade used this PR's chart with two image overrides: agent-server 8bf9c1c-python@sha256:9e2aac… (software-agent-sdk#5691) and LiteLLM 1.100.5@sha256:65cdef….

Check 0.71.1 baseline After upgrade
Keycloak schema 26.3.0 26.8.0, migrated with 0 ERROR lines
Keycloak realm data (realms, users, IdPs, GitHub-linked user, clients) recorded unchanged
App data (users, orgs, conversations, LLM settings, LiteLLM team/token, runtime-api) recorded unchanged
Images bitnamilegacy Keycloak/Postgres/Redis, LiteLLM v1.92.0, Laminar :latest, enterprise-server 1.64.0, agent-server 1.49.5 third-party-images Keycloak/Postgres/Redis, LiteLLM 1.100.5, Laminar 0.1.825, enterprise-server 1.64.0-r1, runtime-api 0.10.0-r1, agent-canvas 1.23.0-r1, sandbox on agent-server digest 9e2aac…
Fresh GitHub login works works
New conversation, agent replies pass pass on agent-server 9e2aac…; also passed earlier on the PR's own pin 7c52a1… (973f316)
Laminar traces — the conversation's trace shows in Laminar, on both agent-server builds

First helm upgrade was marked failed; re-running the same command succeeded. This is the same budget-reconcile-gate race as on kind. On GKE the hook ran while the old openhands-postgresql-0 was still terminating (it took 77s), so the hook got connection refused, and with backoffLimit: 0 the release was marked failed.

Login is down for about 4 minutes during the upgrade. keycloak-0 waits for Postgres, then runs the 26.3 → 26.8 migration. Until it's Ready, Traefik returns no available server on the auth host.

Sandbox image label: same as on kind. The sandbox pod spec shows agent-server:1.49.5-python@sha256:9e2aac…. The digest is the 8bf9c1c image and the running imageID confirms it, but the tag in the name is the old default.


This comment was drafted by an AI agent on behalf of the user.

…s and runtime-api

agent-server 1.49.5-r1-python, agent-canvas 1.23.0-r1 and runtime-api 0.10.0-r1 now point at the release-tag builds instead of the branch candidates. enterprise-server 1.64.0-r1 is a retag of the already-pinned candidate, so its digest is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@dylan-openhands
dylan-openhands marked this pull request as ready for review October 14, 2026 21:08
@dylan-openhands
dylan-openhands merged commit 0361df1 into release/0.71 Oct 14, 2026
27 checks passed
@dylan-openhands
dylan-openhands deleted the dj/c3-0713-chart branch October 14, 2026 22:05
@openhands-release-bot openhands-release-bot Bot added the released: openhands/0.71.3 Shipped in openhands/0.71.3 label Oct 14, 2026
@openhands-release-bot

Copy link
Copy Markdown
Contributor

🚀 Released in openhands/0.71.3.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

released: openhands/0.71.3 Shipped in openhands/0.71.3 type: fix A bug fix

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants