This page owns install, upgrade, uninstall, values, the production render
contract, horizontal scaling, and troubleshooting for the helm/agentflow
chart that deploys the FastAPI API to Kubernetes. Read it when a Kubernetes
operator is installing or upgrading that chart. It does not replace the
shorter curated path in deployment.md.
Audience: Kubernetes operator installing or upgrading the chart
Prerequisites: Kubernetes 1.27+, Helm 3.x, a published or cluster-loaded API image, metrics-server for CPU-based HPA, and a storage class when persistence is enabled; see Prerequisites
For the shorter curated path, start with the deployment walkthrough. This page is the operator reference for the complete Helm contract: chart prerequisites, values, production render clauses, scaling and transport requirements, rollout, upgrade, and troubleshooting. Current deployment evidence and external gates belong to engineering status.
The AgentFlow Helm chart deploys the FastAPI API to Kubernetes with:
- a rolling-update
Deployment - a
Serviceon port8000 - a
PersistentVolumeClaimfor DuckDB files - a
HorizontalPodAutoscalerdriven by CPU - optional
Ingresswith TLS - mounted config files for tenants, SLOs, API versioning, and security policy
- mounted secrets for the admin key and API-key config, either rendered by the chart or supplied through an existing Kubernetes Secret
The chart lives in helm/agentflow.
- Kubernetes 1.27+
- Helm 3.x
- metrics-server if you want the CPU-based HPA to scale automatically
- A container image for the API, published or loaded into the target cluster
- Storage class support if
persistence.enabled=true
The chart deploys the API only. Redis, Kafka, Prometheus, Grafana, Jaeger, and other supporting services stay external to this chart.
The default image.repository is ghcr.io/brownjuly2003-code/agentflow-api,
the registry this project actually publishes to, with image.tag tracking
Chart.appVersion. That tag is a shape, not a pullable reference: the container
workflow pushes commit-SHA and audit-<run-id> tags, never semver ones. So a
dev install does one of two things — build locally and load the image, or
override image.repository and image.tag to wherever your image lives.
# Build locally, then load it under whatever name you set image.repository to:
minikube image load agentflow-api:dev
helm install agentflow helm/agentflow \
--set image.repository=agentflow-api --set image.tag=devThe default deliberately does not point at a bare Docker Hub namespace. It used
to name agentflow/api, which nobody owns: with pullPolicy: IfNotPresent, an
install without a pre-loaded image would have pulled whatever a third party had
pushed there.
Production renders require image.digest; when present, every API-derived
Deployment and provision Job uses repository@digest and ignores the tag.
Use a dedicated values file for production secrets and tenant configuration:
# values-prod.yaml
image:
repository: registry.example.com/agentflow/api
tag: "2.1.0" # required fallback shape, ignored when digest is set
digest: "sha256:REPLACE_WITH_64_LOWERCASE_HEX"
secrets:
create: false
existingSecret: agentflow-api-runtime-secret
config:
tenants:
tenants:
- id: acme-corp
display_name: "Acme Corp"
kafka_topic_prefix: "acme"
max_events_per_day: 1000000
max_api_keys: 10
allowed_entity_types: nullA tenant is isolated by the tenant_id column in each serving table's write key
(ADR-004), so there
is nothing to provision per tenant and nothing further to declare here. The
duckdb_schema field this block used to carry named the old schema-per-tenant
mechanism; the chart still accepts it so existing values keep validating, but
the runtime ignores it.
The existing Secret must contain admin-key and api_keys.yaml.
api_keys.yaml must use the same structured shape as config/api_keys.yaml.
Install the release:
helm install agentflow ./helm/agentflow -f values-prod.yamlThe chart creates separate API, worker, provision, and Flink ServiceAccounts by
default; serving-bridge and lake-materializer reuse the non-privileged worker
identity. Only Flink receives Kubernetes RBAC and a mounted API token. When
serving.backend=clickhouse and provision.enabled=true, the provision
ServiceAccount is an earlier pre-install hook (weight -10, Job is -5) so
first install remains self-contained. Set serviceAccount.apiName,
workerName, provisionName, or flinkJob.serviceAccount to use explicit
role identities. The legacy serviceAccount.name escape hatch assigns one
shared identity to every role and disables this isolation. With
serviceAccount.create=false, all selected accounts must already exist.
Quick dev install with only the admin key overridden, intentionally leaving API
keys empty until you mount or render api_keys.yaml:
helm install agentflow ./helm/agentflow --set secrets.adminKey=local-admin-keyThe chart defaults are dev posture on purpose: no NetworkPolicy, no ingress
TLS, plaintext ClickHouse, an inline Secret. That is what makes the demo
install a single command, and it is the wrong shape for a production release.
config.profile=production is where the two part ways -- it is the operator
declaring this is a production release, and from that point the values are
held to a contract.
helm/agentflow/values-production.yaml
carries the compliant posture and is versioned with the chart. Layer your
environment values on top of it:
helm upgrade --install agentflow ./helm/agentflow -f helm/agentflow/values-production.yaml -f values-prod.yamlThe overlay leaves empty exactly the values only your environment can supply
(the verified API image digest, the Secret name, the origins, the ingress
class/hosts/TLS, the trusted proxies, the Prometheus scrape namespace, the two
pepper extraEnv entries and the egress destinations described below).
Helm replaces lists instead of merging them — an environment file that
names only the scrape namespace removes the ingress-controller entry and
blocks the production Ingress. Repeat both selectors in the environment
file:
networkPolicy:
ingressFromNamespaces:
- kubernetes.io/metadata.name: ingress-nginx
- kubernetes.io/metadata.name: monitoringRender is fail-closed:
templates/production-contract.yaml refuses a
profile=production render that still violates the contract and reports every
violation in one message, so you fix the whole set in one pass. It checks:
| Clause | Why |
|---|---|
image.digest=sha256:... |
Every API-derived workload consumes one immutable artifact; a tag cannot prove staging/release identity |
networkPolicy.enabled=true |
Default-deny baseline; needs a NetworkPolicy controller in the cluster |
networkPolicy.ingressFromNamespaces not empty; not every kubernetes.io/metadata.name equal to ingress-nginx |
/metrics is unauthenticated for in-cluster scrape; the NetworkPolicy is the allow-list. An empty list renders ingress: [] (deny all), so neither the ingress controller nor Prometheus can reach the service port. An empty-map entry ({}) matches every namespace and is refused. A non-empty list is refused when every entry that carries kubernetes.io/metadata.name equals ingress-nginx and no non-empty entry lacks that key (a non-empty entry selecting by another label counts as other and passes), unless networkPolicy.scrapeFromIngressNamespace=true records that Prometheus deliberately runs in the ingress-controller namespace |
service.type=ClusterIP |
NodePort/LoadBalancer publish the service port (/metrics included) without any Ingress rule; ExternalName turns the Service into a CNAME and voids the routing contract |
secrets.create=false + existingSecret |
Values persist in Helm release metadata and shell history |
Empty secrets.adminKey / apiKeys.keys |
Inline key material is dev-only |
extraEnv sets AGENTFLOW_KEY_LOOKUP_PEPPER and AGENTFLOW_QUERY_FINGERPRINT_PEPPER, each via valueFrom.secretKeyRef |
Both peppers fall back to constants committed to this repository, and the app refuses to boot on production without them. A literal value: is refused for the same reason secrets.create=true is |
networkPolicy.egressTo.<service> non-empty for every egress rule that renders |
An egress rule with ports: and no to: allows that port to every address, in the cluster and on the internet, so the default-deny baseline denies nothing there |
ingress.hosts non-empty when ingress is enabled |
An Ingress with no rules routes nothing |
Every ingress.hosts[] entry has a non-empty paths list |
An Ingress rule with no HTTP paths is invalid and routes nothing |
ingress.tls non-empty when ingress is enabled |
TLS terminates somewhere you can point at |
ingress.annotations has none of rewrite-target, use-regex, app-root, configuration-snippet, or server-snippet under nginx.ingress.kubernetes.io/ or legacy ingress.kubernetes.io/ |
ingress-nginx interprets these routing-control annotations after Helm checks the literal host/path, so they can change matching or the upstream URI and reach /metrics |
ingress.hosts[].paths[] must not route /metrics |
/metrics is unauthenticated for in-cluster scrape; enumerate the public prefixes instead of Prefix / (Production ingress and /metrics) |
pathType is Prefix or Exact |
Controller-defined matching cannot be proven at render time, so ImplementationSpecific cannot be shown to keep /metrics off Ingress |
path matches ^/[A-Za-z0-9._~!$&'()*+,;=:@/-]*$; host and className match ^[A-Za-z0-9*]([A-Za-z0-9.-]*[A-Za-z0-9])?$ |
Canonical single-line routing values exclude whitespace, YAML injection, and controller-ambiguous spellings that could expose unmatched /metrics |
config.trustedProxies non-empty, or config.gateway.preservesClientIp=true |
Behind any proxy -- this chart's ingress or a gateway outside it -- every caller otherwise shares that proxy's address, which is what the failed-auth limiter keys on and what every logged client_ip reports. ingress.enabled=false no longer makes the question disappear: name the peers, or declare that the path preserves the caller's source address |
Explicit config.corsOrigins |
CORS runs with credentials; a wildcard lets any site read authenticated responses, and the chart's localhost default is not an answer |
serving.clickhouse.secure=true |
No plaintext hop to an external ClickHouse |
config.redisUrl on rediss:// |
Same, for Redis |
Empty serviceAccount.name |
The shared-identity escape hatch disables RBAC isolation |
config.security on argon2id with all five redacted headers |
The mounted policy must not be weaker than config/security.yaml |
Two escapes are deliberate rather than accidental. A plaintext hop that is a
considered decision -- in-cluster traffic behind a NetworkPolicy, say -- is
named per store in AGENTFLOW_INSECURE_TRANSPORT_OK through extraEnv, the
same greppable opt-out the runtime honours at boot. And terminating TLS in a
gateway ahead of the chart is a legitimate topology: set ingress.enabled=false
and the TLS and trusted-proxy clauses stop applying. That move puts the routing
decision, and the /metrics exposure question, outside this chart.
The annotation clause is an exact denylist on the Ingress object rendered by
this chart. It does not constrain the ingress-nginx controller ConfigMap or
separately managed Ingress objects; review those cluster-level inputs as part
of the platform routing policy. Unrelated annotations such as
cert-manager.io/cluster-issuer, nginx.ingress.kubernetes.io/ssl-redirect,
and nginx.ingress.kubernetes.io/proxy-body-size remain available.
Enforced elsewhere, so the contract does not repeat it: Kafka SASL/TLS for
every Kafka workload (templates/_kafka.tpl), plaintext external stores at boot
(agentflow_runtime.serving.transport_policy -- including the control-plane
DSN, which the chart only ever sees as a Secret reference), and multi-replica
gating on an external serving engine plus an external control plane
(ADR-0010).
The protected build-and-sign job in
.github/workflows/container-attestation.yml owns the image build. After it
pushes, signs, and emits build provenance for that digest, it uploads
agentflow-image-promotion-<git-sha>. The artifact contains:
image-values.yamlwith the exact repository and digest;helm-deployment.yaml, rendered from that values file with pinned Helm;promotion.jsonwith the Git SHA, workflow run ID, immutable image subject, tool versions, and SHA-256 of the rendered Deployment.
The packet proves that Helm can consume the image built by that workflow
without rebuilding it. staging-deploy.yml is its manual, fail-closed
consumer. An operator supplies the successful build run ID, its exact
main-branch source SHA, and PROMOTE. Before kind starts, the workflow:
- checks the selected workflow run and
build-push-sign-attestjob through the Actions API; - downloads the source-SHA-named artifact from that run and validates the packet schema, filenames, identities, values, manifest, and checksums;
- verifies the exact OCI subject with cosign and GitHub SLSA provenance;
- passes only the verified digest values to Helm, without
docker buildorkind load.
The existing smoke and E2E suites remain the rollout gate. After they and
staging teardown pass, the workflow uploads
agentflow-staging-promotion-<source-sha>-<staging-run-id> with checksummed
signature/provenance results and staging evidence. This proves only that the
selected digest passed staging. It does not prove a production rollout,
production acceptance, or complete F-19 closure.
Check the release status:
kubectl get pods -l app.kubernetes.io/instance=agentflow
kubectl rollout status deployment/agentflow
kubectl get hpa,pvcPort-forward the service and call the health endpoint:
kubectl port-forward svc/agentflow 8000:8000
curl http://127.0.0.1:8000/v1/healthIf ingress.enabled=true, verify the configured host instead of using port-forwarding.
config.duckdbPathandconfig.usageDbPathshould point to the mounted PVC path.config.contractsDirpoints at contract YAML files bundled into the image. The chart does not mountconfig/contracts/separately.secrets.apiKeys.keys[*].key_idis required for deterministic admin rotation and staging checks.- Default
secrets.apiKeys.keysis empty. Supply API-key config throughsecrets.existingSecretor through an environment-specific values file; do not reuse repository defaults as runtime credentials. - If
secrets.create=false,secrets.existingSecretmust name a Kubernetes Secret withadmin-keyandapi_keys.yaml. - The default chart mounts that Secret at
config.apiKeysPath(/etc/agentflow/secret/api_keys.yaml) read-only. Mutable key lifecycle (POST /v1/admin/keys, rotate, revoke-old,DELETE /v1/admin/keys/{key_id}) needs a writable durable store. Calling those endpoints against the default mount returns409 Conflictwith a secret-free detail. Multi-replica key mutation also needs a shared store; a per-pod file would diverge even if the volume were writable. config.tenantsis the source of truth for tenant routing and API version pinning.autoscaling.enabled=truecreates an HPA fromminReplicastomaxReplicas, but persistent DuckDB storage is guarded to one writer replica. Rendering fails whenpersistence.enabled=trueand the chart is configured for more than one API writer replica. Multi-replica also requires the external control-plane store — see Horizontal scaling.ingress.tlsaccepts the standard Helm ingress TLS structure.- ConfigMap and Secret checksums are injected into the pod template, so
helm upgraderolls the deployment when mounted config changes. - DuckDB is still a stateful local file. If your storage class only supports
ReadWriteOnce, start withreplicaCount: 1until you validate your storage and concurrency model. - Optional DuckDB file encryption is runtime-configured with
AGENTFLOW_DUCKDB_ENCRYPTION_KEYorAGENTFLOW_DUCKDB_ENCRYPTION_KEY_FILE; useextraEnvwith asecretKeyRefto supply the key. The default remains unencrypted for backward compatibility. - DuckDB encryption is a local at-rest hardening option only. It is not a NIST, GDPR, HIPAA, SOC 2, or external-compliance attestation by itself.
- Optional append-only audit export is runtime-configured with
AGENTFLOW_AUDIT_LOG_PATH, which writes a hash-chained JSONL file in addition to DuckDB usage analytics. For externally immutable retention, operators still need object-lock or SIEM evidence outside this chart.
The chart default is the single-replica, zero-dependency demo (DuckDB serving, embedded per-pod control plane). Scaling the API horizontally needs both halves of the ADR 0009/0010 gate, enforced at render time — the chart fails any multi-replica render unless both are set:
- External serving engine —
serving.backend=clickhouse(+serving.clickhouse.hostand a passwordexistingSecret). ADR 0006/0007. - External control-plane store —
controlPlane.store=postgreswith a DSN incontrolPlane.postgres.existingSecret(keycontrolPlane.postgres.dsnKey, defaultcontrolplane-pg-dsn). ADR 0009/0010. This moves the webhook queue/registrations, alert rules+history, outbox/dead-letter and usage out of the per-pod DuckDB/YAML into one shared store, so N replicas no longer fork that state (duplicate deliveries, split alert history).
The chart ships no PostgreSQL and no ClickHouse service, exactly as it
consumes an external ClickHouse: the operator provides both and the referenced
secrets. The DSN carries a password, so — like the ClickHouse password — it is
never inlined into values, only referenced via existingSecret.
The scale profile crosses real network boundaries, so transport security is
first-class in values, not an extraEnv afterthought:
- ClickHouse —
serving.clickhouse.secure=trueswitches the client to HTTPS with certificate and hostname verification (the server certificate must matchserving.clickhouse.host). For a private CA, create a Secret holding the PEM bundle and setserving.clickhouse.tls.caSecret(key name intls.caKey, defaultca.crt); it is mounted read-only into both the API pods and the provision Job, andCLICKHOUSE_CA_CERTthen replaces the system trust store for that connection. Client certificates are not supported by the chart; terminate mTLS at your ingress/mesh if required. - PostgreSQL — put
sslmode=require(orverify-ca/verify-full) in the DSN stored incontrolPlane.postgres.existingSecret. - Redis — use a
rediss://URL inconfig.redisUrl.
config.profile=production arms the app-side gate: the boot fails when an
external ClickHouse/Redis/PostgreSQL hop is plaintext (loopback is exempt), and
the demo surface refuses to come up at all. A deliberate plaintext hop — e.g.
in-cluster traffic already constrained by the chart's NetworkPolicy — must be
named explicitly via extraEnv:
AGENTFLOW_INSECURE_TRANSPORT_OK="clickhouse,redis". A wildcard
config.corsOrigins is likewise refused outside demo mode.
policyTypes: [Ingress, Egress] makes the NetworkPolicy an allow-list in both
directions, but a Kubernetes egress rule with ports: and no to: allows that
port to every address — other namespaces, the node network, the internet.
The chart used to render exactly that for Redis, Kafka, Iceberg, the object
store, ClickHouse, OTLP and PostgreSQL, so the baseline denied nothing on
6379/9092/8181/9000/8123/4317/5432 (audit FB-09).
Each rule now takes its peers from networkPolicy.egressTo.<service>, as raw
NetworkPolicyPeer entries — podSelector, namespaceSelector, ipBlock, or
a combination:
networkPolicy:
egressTo:
kafka:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: data
podSelector:
matchLabels:
app.kubernetes.io/name: kafka
postgres:
- ipBlock:
cidr: 10.30.4.0/24 # managed PostgreSQL outside the clusterA rule only renders when its feature is configured, and the contract asks only
for the rules that render. On the chart defaults — DuckDB, no config.redisUrl,
no config.otlpEndpoint, the embedded control plane, lakeMaterializer off —
that is Kafka alone; switching on ClickHouse, Redis, OTLP, PostgreSQL or the
lake materializer adds its own. DNS is the one rule the chart owns outright: it
selects kube-dns by label in whichever namespace it runs.
Two environment variables domain-separate digests the API stores, and both fall back to constants committed to this repository:
| Variable | Peppers | Left at the default |
|---|---|---|
AGENTFLOW_KEY_LOOKUP_PEPPER |
key_lookup, an HMAC-SHA256 of the API key itself, stored beside the argon2id hash so authentication resolves the candidate in O(1) |
A leaked api_keys.yaml lets a guessed key be confirmed against the stored digest with one HMAC and no argon2id verify, and lets two deployments' digests be joined into one identity (audit FB-07) |
AGENTFLOW_QUERY_FINGERPRINT_PEPPER |
Query-analytics fingerprints | A leaked analytics table can be joined against question digests computed anywhere else (audit AF-13) |
config.profile=production refuses to boot with either unset or left at its
default, and the production contract refuses the render before that. Both are
supplied through extraEnv, projected from the Secret your secrets operator or
CSI driver manages — never as a literal value:, which would park the pepper in
Helm release metadata and in the shell history of whoever ran the upgrade:
extraEnv:
- name: AGENTFLOW_KEY_LOOKUP_PEPPER
valueFrom:
secretKeyRef:
name: agentflow-production-secret # your secrets.existingSecret
key: key-lookup-pepper
- name: AGENTFLOW_QUERY_FINGERPRINT_PEPPER
valueFrom:
secretKeyRef:
name: agentflow-production-secret
key: query-fingerprint-pepperHelm replaces lists rather than merging them, so this block must carry every
extraEnv entry the release needs, AGENTFLOW_INSECURE_TRANSPORT_OK included.
Rotating the lookup pepper is not free: it invalidates every stored
key_lookup, and authentication falls back to the O(n) verify scan until the
keys are re-issued. See
docs/runbooks/auth-401-spike.md.
k8s/staging/values-staging-scale.yaml.example is a ready overlay:
helm upgrade --install agentflow ./helm/agentflow \
-f k8s/staging/values-staging.yaml \
-f k8s/staging/values-staging-scale.yaml \
--namespace agentflowReplica-correctness is verified by scripts/k8s_replica_correctness_verify.sh
(Checks 1–4: ≥2 postgres pods, cross-pod registration visibility, exactly-one
delivery per (webhook, event) via shared ClickHouse journal insert, and exactly
one alert page per incident via claim_alert_tick). See
docs/plans/clickhouse-cutover-plan.md Phase 3. With the postgres store the render
gate relaxes automatically; with controlPlane.store=embedded any
replicaCount > 1 (or autoscaling.maxReplicas > 1) render is refused.
helm/agentflow/values.schema.jsonis the chart contract for runtime values consumed from Helm.- If you add, rename, or make required a field under
config.tenantsorsecrets.apiKeys, update the schema, chart defaults, and environment-specific values together. - Keep the mounted file shape in
templates/configmap.yamlandtemplates/secret.yamlaligned with the runtime Pydantic models insrc/agentflow_runtime/tenancy.pyandsrc/agentflow_runtime/serving/api/auth/manager.py. - Validate contract changes with
helm lint helm/agentflow -f k8s/staging/values-staging.yamlbefore staging rehearsal.
Update the values file or image tag, then run:
helm upgrade agentflow ./helm/agentflow -f values-prod.yaml
kubectl rollout status deployment/agentflowThe deployment strategy uses maxUnavailable: 0 and maxSurge: 1 to avoid downtime during a normal rolling update.
Remove the release:
helm uninstall agentflowIf you also want to remove persisted DuckDB data, delete the PVC after the uninstall:
kubectl delete pvc agentflowImagePullBackOff: for dev, setimage.repositoryandimage.tagor load the image into Minikube; for production, verify thatimage.digestexists in that repository.Pendingpod with PVC errors: choose a validpersistence.storageClassNameor disable persistence for ephemeral environments.503on admin endpoints: setsecrets.adminKey.- Missing auth or tenant config: check the rendered
api_keys.yamlandtenants.yamlvalues inside the mounted Secret and ConfigMap.