TLS handshake failures when updating from 1.9.0 to 1.9.1 (related to #9519)

[lboynton]

Description:
During an upgrade from Envoy Gateway 1.9.0 to 1.9.1, we noticed a spike in TLS handshake errors. Root cause is the same class of bug as #9519 — pod readiness/liveness never checks whether SDS has actually delivered certificates — but via a different trigger: a brand-new pod's very first SDS fetch on cold start, racing a mass simultaneous pod restart, rather than #9519's stale delta-ADS re-subscription on a long-running proxy. Filing separately since the trigger and recovery profile are meaningfully different, but linking as clearly related.

Once the upgrade rollout had finished the TLS handshake errors went away.

Failure mechanism:

1. Trigger — routine version upgrade. We applied the Envoy Gateway 1.9.0 → 1.9.1 update to our cluster. This rolled all proxy pods to a new ReplicaSet.
2. Mass simultaneous cold start. All new pods started at once, each needing to fetch ~53 distinct SDS secrets (TLS certs for our configured Gateways/HTTPRoutes) from the envoy-gateway control plane simultaneously — i.e. ~477 concurrent initial SDS fetches hitting the control plane at once.
3. Readiness passes independent of SDS state. Per internal/xds/translator/listener_ready.go and internal/xds/filters/wellknown.go, the readiness listener uses the stock health_check HTTP filter with PassThroughMode: false — it answers purely from the Envoy process's own liveness flag, with no dependency on LDS/SDS convergence. So every pod passed Kubernetes readiness almost immediately, well before its 53 SDS secrets had loaded, and was added to the Service endpoints.
4. Real traffic hits listeners with SDS still outstanding. With the mass fetch contending against the control plane, a meaningful number of those initial fetches took long enough to hit the (correctly, per 1.9.1's fix) reverted 15s initial_fetch_timeout for SDS. Per GrpcSubscriptionImpl::onConfigUpdateFailed (Envoy core), a timed-out initial fetch still unblocks the init manager — so the listener becomes "active" without ever having received a valid cert for the affected filter chain. Live TLS handshakes against that pod failed until the pod's retry succeeded.
5. Self-healing, unlike Proxies permanently lose TLS after a listener/cluster update that follows an Envoy Gateway restart — new SDS subscriptions are never fetched over delta ADS (15s initial_fetch_timeout activates listeners without certs) #9519. Each pod's own retry-after-timeout eventually succeeded once control-plane contention eased.

Repro steps:

1. Install Envoy Gateway v1.9.1+ with a Gateway/HTTPRoute configuration referencing a non-trivial number of TLS secrets — we saw this reliably with ~50 per proxy pod; a handful may not create enough control-plane fetch contention to lose the race, so scale up if it doesn't reproduce at low secret counts.
2. Send steady TLS traffic to the Gateway's listener(s) so there's something to observe failing.
3. Trigger a rollout of all proxy pods at once so many pods request their initial SDS secrets from the control plane at the same moment.
4. While the rollout is in progress, in parallel:
5. Tail the new pods' logs for gRPC config: initial fetch timed out for type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.Secret.
6. Poll each new pod's readiness endpoint (the port/path configured via bootstrap.EnvoyReadinessPort/EnvoyReadinessPath, exposed at the Kubernetes readiness probe) — it will report healthy immediately, before and during the SDS timeout warnings.
7. Send real TLS requests directly at each new pod's Service IP (bypassing any external LB retry/failover) as it becomes "Ready," to observe handshake failures independent of upstream retry logic masking the issue.
8. Confirm recovery: once a pod's SDS retry succeeds, handshakes against it succeed again, without a pod restart — this distinguishes it from Proxies permanently lose TLS after a listener/cluster update that follows an Envoy Gateway restart — new SDS subscriptions are never fetched over delta ADS (15s initial_fetch_timeout activates listeners without certs) #9519, where recovery required a manual restart.
9. To isolate this from Proxies permanently lose TLS after a listener/cluster update that follows an Envoy Gateway restart — new SDS subscriptions are never fetched over delta ADS (15s initial_fetch_timeout activates listeners without certs) #9519's trigger, do not restart the envoy-gateway control-plane pod or otherwise disturb an already-running proxy's stream — this repro is specifically about a fresh pod's very first fetch, not a re-subscription on a warm one.

Environment:
Envoy Gateway 1.9.1

Logs:
Proxy log (representative — reconstructed from Envoy's source-level format at source/extensions/config_subscription/grpc/grpc_subscription_impl.cc:130 plus our aggregated log-count evidence; we have counts per pod via our log platform, not a saved verbatim capture of a single raw line):

[2026-08-28 14:08:2X.XXX][1][warning][config] [source/extensions/config_subscription/grpc/grpc_subscription_impl.cc:130] gRPC config: initial fetch timed out for type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.Secret

This line repeated exactly 53 times in immediate succession per new pod — once per configured TLS secret — within the first couple of seconds after each pod's readiness probe had already started reporting healthy.

[lboynton]

Correction to the trigger described above

Edited shortly after posting to fix a mechanism error below — see the note in "Relationship to #9519" for what changed.

We've since re-investigated using metrics rather than just logs, and the "fresh pod cold start" trigger described above doesn't match what actually happened. Correcting the record here, since it also affects how this issue relates to #9519.

What we originally believed: the 1.9.0 → 1.9.1 upgrade rolled all proxy pods to a new ReplicaSet, and the 9 pods that logged SDS initial fetch timed out were those brand-new pods hitting contention on their very first fetch.

What we now have evidence for: those pods never restarted at all.

• kube_pod_container_status_restarts_total was flat at 0 for all of them across the incident window.
• container_start_time_seconds showed each had been running continuously since well before the incident — in several cases, over a day.
• A genuine proxy-pod rollout was happening at the same time (the 1.9.0 → 1.9.1 image bump did land, via a normal, separate, slower rolling deployment), which is what led us to conflate the two. But the specific pods that logged SDS timeouts were old-generation pods that hadn't been replaced yet, not new ones.

What actually restarted was our envoy-gateway controller — all 3 replicas, within ~13 seconds of each other, as part of the same Helm upgrade. The affected proxies were long-running, already-Ready, already-initialized processes that lost their xDS connection when the controller fleet went down underneath them, and re-established it once the new controller pods came up.

Why that matters for the mechanism, not just the timeline: a genuinely cold-starting pod's SDS provider has no previously-agreed resource version — there's nothing for a version bump to invalidate, and the fail-open/timeout-but-watch-stays-registered behavior in GrpcSubscriptionImpl is enough on its own to explain a self-heal once contention drains. That's a real, distinct, self-healing failure mode, and it may be worth its own issue if anyone wants to pursue it (mass simultaneous cold start under high secret-per-pod counts, initial_fetch_timeout restored to a real 15s in 1.9.1) — but it isn't what happened here.

Our proxies were warm, not cold, and per a closer look at how delta xDS actually diffs, that turns out to matter more than we first thought. GrpcSubscriptionImpl's init-fetch-timeout timer is created exactly once, in start(), and is disabled and released — not re-armed — once that subscription's first update lands; reconnecting the underlying gRPC stream doesn't recreate the per-resource subscription object or its timer. So a warm, already-satisfied SDS watch genuinely can't produce a fresh "initial fetch timed out" from a bare stream reconnect. For our warm pods to log real SDS timeouts, new SDS provider objects must have been created on them after the controller restart — the same structural precondition described below for #9519 (a provider-cache-key miss on hash(sds_config_source) + name while an old watch on that name is still alive), not a distinct "genuine active resync" mechanism. (An earlier version of this comment attributed the self-heal to non-content-stable xDS snapshot versioning forcing a full resend on reconnect — that doesn't hold up: delta xDS diffs by per-resource content hash, not by the snapshot's own version string, so an unchanged secret produces no resend regardless of whether the version counter reset. Retracting that explanation.)

We don't yet have direct evidence of why a new provider would get created on a proxy that never itself restarted — a listener/cluster update pushed by the newly-restarted controller with byte-different embedded SDS config-source references is our leading candidate, but that's inference, not confirmed. envoy_sds_update_attempt on the affected pods increments multiple times during the incident window rather than sitting flat (e.g. one pod steps 4→5→6→7 across its ~70s failure window), which at least rules out a totally silent, zero-wire-activity pattern — though we've since learned it doesn't prove much beyond that: that stat's scope is keyed purely by secret name (sds.<name>.), so an old and a new provider for the same secret share the same counter, and it increments on several call sites (watch-interest changes, on-demand requests, both successes and failures) rather than only on confirmed wire sends.

Relationship to #9519, corrected (three times now): we'd previously drawn the comparison assuming our case was a cold-start variant of the same readiness gap, then revised that to a "genuine resync via version mismatch" story, then to "the starved provider's own request eventually got through once a cache key freed up." All three need correcting. The last one has a mechanical hole: a starved watch is never itself the thing sending a request — per WatchMap::findAdditions, it was silently registered on the existing entry and never triggered a subscribe in the first place, so nothing about another provider's teardown changes that. What actually un-starves it: WatchMap's fan-out (watchesInterestedIn) delivers any incoming push of that resource to every registered watch, including one that never sent its own request. So a starved provider heals passively, the moment the control plane pushes that secret again for any reason — it doesn't need to have asked. As best we can currently tell, there are two distinct mechanisms here, sharing both the EG-level readiness/SDS-convergence gap and the provider-cache-key precondition:

1. This incident: warm proxy, controller-fleet restart, a new SDS provider gets created for a secret an old, still-alive watch already covers (exact trigger for the new provider not yet confirmed) — the same structural precondition Proxies permanently lose TLS after a listener/cluster update that follows an Envoy Gateway restart — new SDS subscriptions are never fetched over delta ADS (15s initial_fetch_timeout activates listeners without certs) #9519 describes. Ours self-healed because something re-pushed those secrets during the window (plausible amid 3 controllers restarting plus a proxy image rollout running in parallel) — the starved watch was fed by the fan-out without ever needing its own request answered.
2. A genuine cold-start variant (as originally, incorrectly, described above): brand-new pod, no prior version, fail-open timeout, watch stays registered, self-heals once contention drains. A real, distinct, likely self-healing failure mode — just not what happened here.
3. Proxies permanently lose TLS after a listener/cluster update that follows an Envoy Gateway restart — new SDS subscriptions are never fetched over delta ADS (15s initial_fetch_timeout activates listeners without certs) #9519: the same provider-cache-key-miss precondition as (1), but — per Proxies permanently lose TLS after a listener/cluster update that follows an Envoy Gateway restart — new SDS subscriptions are never fetched over delta ADS (15s initial_fetch_timeout activates listeners without certs) #9519's own report — the secret itself was never re-pushed for any reason after the reconnect, so the fan-out that rescued ours never fired for them. Same starved state, permanently, until a manual restart.

Given (1) and (3) now look like the same underlying mechanism rather than distinct ones, we don't think there's a clean "only #9519 needs the upstream fix, ours is fully EG-side" split anymore — both may hinge on the same upstream question about why a new provider gets created at all on a proxy that never restarted. We'd rather get that pinned down than keep revising the comparison in place; happy to share the Prometheus queries either way.

Apologies for the repeated correction on this thread.

[dnmgns]

We also hit this issue during a routine v1.9.0 to v1.9.1 upgrade and can confirm it's not tied to high SDS cardinality specifically. Each of our Gateways terminates TLS with a single wildcard certificate (1 SDS secret per proxy pod), so the "~50 concurrent secrets" scenario in the original report doesn't seem like a precondition, rather just an amplifier.

What we observed: the envoy-gateway controller and the data-plane Deployment rolled onto the new version at (essentially) the same time. The controller reconciled and regenerated the proxy bootstrap on startup, which immediately triggered the data-plane rollout. New proxy pods passed the container's /ready probe and were added to our NLB target group before they'd received a fresh LDS/SDS push from the just-restarted controller, causing TLS handshakes to abort client-side (PR_END_OF_FILE_ERROR in Firefox) for a short window (~1-2 min) before it self-healed.

I believe the fix in this case needs to be in the readiness check itself, gating on LDS/SDS convergence rather than process liveness.

Happy to share more detail if useful for triage!

[lboynton]

Reproducing #9918 — EG 1.9.0 → 1.9.1 upgrade starves every SDS provider on warm proxies

Summary. Envoy Gateway v1.9.1 removed source.InitialFetchTimeout = durationpb.New(0) from
makeConfigSource() (commit c30c739f8, the v1.9.1 cherry-pick). That single line does two
things at once:

1. It flips a Duration field from present to absent on every generated SDS config
   source, which changes MessageUtil::hash(sds_config_source) — the SecretManager's provider
   cache key. On upgrade, warm proxies get listeners that miss the cache for every secret while
   the old providers are still alive holding the xDS watches, so
   WatchMap::findAdditions suppresses every wire request.
2. It removes the initial_fetch_timeout: 0 protection, reverting to Envoy's 15s default —
   re-arming the very timer that turns silent starvation into an outage.

Net effect on a proxy that never restarted: N SDS initial-fetch timeouts (N = number of TLS
secrets) ~15s after the new controller pushes listeners, listeners go active without
certificates, Kubernetes readiness stays green, and all TLS handshakes fail.

Under 1.9.0 the identical cache-key miss was invisible: with 0s the provider waits forever, the
new listener stays warming, and the old one keeps serving. That is why this only appears on
upgrade.

---

Fast confirmation (no cluster required)

git diff v1.9.0..v1.9.1 -- internal/xds/translator/listener.go | grep -A8 makeConfigSource

source.InitialFetchTimeout = durationpb.New(0) present in v1.9.0, gone in v1.9.1.

Confirm the emitted bytes actually change:

git diff v1.9.0..v1.9.1 -- \
  internal/xds/translator/testdata/out/xds-ir/http-route-https-mtls-accept-untrusted.listeners.yaml

           sdsConfig:
             ads: {}
-            initialFetchTimeout: 0s
             resourceApiVersion: V3

Blast radius: makeConfigSource() has 11 call sites, including RDS (listener.go:382), OIDC,
and the credential injector. TLS is where it is fatal, but it is not TLS-only.

---

Live reproduction

Preconditions

• Proxies must be warm across the upgrade. A proxy that cold-starts on 1.9.1 fetches its
  secrets normally and is unaffected.
• The TLS Secrets must not change during the observation window, or the starved providers are
  fed immediately and it self-heals.

1. Install EG 1.9.0 with TLS termination

One HTTPS listener and one cert Secret is sufficient — confirmed experimentally (see
"Minimal repro: a single TLS cert" below). Secret count is not a factor in the trigger; it only
determines how many timeout lines you see, since each secret gets its own starved provider.

helm install eg oci://docker.io/envoyproxy/gateway-helm --version v1.9.0 \
  -n envoy-gateway-system --create-namespace

Apply a Gateway with an HTTPS listener referencing a TLS Secret, plus an HTTPRoute and backend.

2. Let it converge, then record the 1.9.0 shape

kubectl port-forward -n envoy-gateway-system <proxy-pod> 19000:19000
curl -s 'localhost:19000/config_dump?resource=dynamic_listeners' \
  | grep -A3 tls_certificate_sds_secret_configs

Expect "initial_fetch_timeout": "0s" on every SDS config source. Confirm TLS serves and every
envoy_sds_init_fetch_timeout is 0.

3. Make the rollout slow, and use several replicas

You cannot prevent the proxy rollout. The proxy pod template embeds the Envoy bootstrap as
--config-yaml, and that bootstrap differs between 1.9.0 and 1.9.1, so the pod template hash
changes on upgrade no matter what you pin. (Pinning envoyDeployment.container.image and
shutdownManager.image to the 1.9.0 values is not sufficient — verified.)

That is fine, because it is also what actually happens during the upgrade: the rollout is gradual,
and the new 1.9.1 controller pushes config to every connected proxy immediately, including
old-generation pods still queued behind the rolling update. Those pods are the subjects.

Give yourself a long observation window:

# EnvoyProxy attached to the Gateway via infrastructure.parametersRef
spec:
  provider:
    kubernetes:
      envoyDeployment:
        replicas: 10
        strategy:
          type: RollingUpdate
          rollingUpdate:
            maxSurge: 1
            maxUnavailable: 0   # replace one at a time

Record the old-generation pod names and IPs before upgrading.

4. Upgrade the control plane

kubectl set image deploy/envoy-gateway -n envoy-gateway-system \
  envoy-gateway=envoyproxy/gateway:v1.9.1

(Or helm upgrade ... --version v1.9.1. The image swap is sufficient — CRDs are compatible within
a patch release, and it avoids needing the release manifests.)

Only count pods that are Running with no deletionTimestamp. A terminating pod shows
ready=200 with dead TLS purely as a graceful-drain artifact — the readiness listener still answers
while the TLS listeners drain. That looks identical to the bug and will fool you.

5. Observe

The new controller regenerates and pushes listeners whose sds_config bytes now differ. Then,
roughly 15s later:

kubectl logs -n envoy-gateway-system <warm-proxy-pod> -c envoy \
  | grep -c "initial fetch timed out for type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.Secret"

Expected, all on a pod with restarts=0:

Signal	Expected
timeout log lines	exactly N (one per TLS secret), ~15s after the push
TLS handshakes	fail on all SNIs
readiness (/ready, port 19003)	200 throughout
listener_manager.total_listeners_active / _warming	active > 0, warming 0
envoy_listener_ssl_connection_error	0
..._downstream_context_secrets_not_ready	climbing
envoy_sds_update_success{...}	does not increment for starved secrets

Trust envoy_sds_update_success, not envoy_sds_update_attempt. update_attempt is inflated
because the old and new provider for a given secret share a stat scope keyed on the secret name
(sds_api.cc createScopeWithTaggedName), and it increments on six different call sites in
grpc_subscription_impl.cc. update_success only increments after a config update is actually
accepted.

6. Confirm the byte flip

Re-run the step 2 config dump. initial_fetch_timeout is now absent.

7. Confirm recovery is per-secret

This is the diagnostic that pins the mechanism. Rotate the contents of one Secret and watch
only that SNI recover — the others stay dead, with no pod restart. A starved provider is fed only
when the control plane pushes that specific secret, via the WatchMap::watchesInterestedIn
fan-out. It never issues a request of its own.

Full recovery: restart the proxy pods (fresh providers, normal fetch).

---

Confirmed result of the two-binary run

Real envoyproxy/gateway:v1.9.0 → v1.9.1 controller image swap, 10 warm proxies, one-at-a-time
rollout:

• 7 of 10 old-generation pods were Running, not terminating, restarts=0 — and each
  logged exactly 55 initial fetch timed out ... tls.v3.Secret lines (one per secret).
• Sample pod ...7b9f777f74-45nf5: envoy started 10:39:06Z, upgrade issued 10:41:28Z, first
  timeout 10:41:45.626Z (~15s after the new controller pushed).
• On that pod: readiness 200, 0/55 SNIs serving, total_listeners_active 3 /
  _warming 0, secrets_not_ready 55, ssl_connection_error 0, 55 secrets with
  init_fetch_timeout > 0.
• Its own config dump flipped from
  {"ads":{},"initial_fetch_timeout":"0s","resource_api_version":"V3"} × 55 (under 1.9.0) to
  {"ads":{},"resource_api_version":"V3"} × 55 — delivered over the wire, on a pod that never
  restarted.

That is the reported incident signature, reproduced end to end with real release binaries.

Minimal repro: a single TLS cert

The same real-binary upgrade was repeated with a single Gateway listener and a single TLS
Secret (single/single-tls), 10 replicas, one-at-a-time rollout:

• All 10 old-generation pods were Running, not terminating, restarts=0, each with
  exactly 1 SDS timeout.
• Sample pod: envoy started 10:52:46Z, upgrade issued 10:54:21Z, timeout 10:54:38.396Z
  (~17s), readiness 200, single.test TLS dead, listeners_active 3 / _warming 0,
  envoy_sds_init_fetch_timeout{envoy_xds_resource_name="single/single-tls"} 1.

So secret count is irrelevant to the trigger — 53 or 55 timeout lines simply reflects config size,
not scale or contention. Any Gateway terminating TLS is affected on upgrade.

Note that xds_certificate and xds_trusted_ca stayed at init_fetch_timeout 0: those use
path_config_source (file-based SDS) rather than ADS, so they are structurally immune. Only
ADS-delivered secrets are affected.

Timing note: with few replicas the rolling update can finish in under a minute and you will miss
the window entirely. Use ~10 replicas with maxSurge: 1 / maxUnavailable: 0, and start a sampler
(every ~2s, recording phase + deletionTimestamp + timeout count per pod) before issuing the
upgrade.

---

Negative controls worth including in a report

• Cold-start pods are unaffected. A proxy starting fresh on 1.9.1 fetches normally. Rules out
  control-plane contention as the cause.
• 1.9.0 alone is unaffected. The same cache-key miss under 0s leaves the new listener
  warming while the old one keeps serving — same starvation, no outage.

---

Relationship to #9519

Same mechanism, different reason for the outcome:

• TLS handshake failures when updating from 1.9.0 to 1.9.1 (related to #9519) #9918 (this): starved providers, but the control plane later re-pushes the secrets, so the
  fan-out feeds them. Self-heals without a restart.
• Proxies permanently lose TLS after a listener/cluster update that follows an Envoy Gateway restart — new SDS subscriptions are never fetched over delta ADS (15s initial_fetch_timeout activates listeners without certs) #9519: the same provider-cache-key miss, but the secret is unchanged after reconnect (its
  own stated third condition), so nothing ever pushes it again. Nothing to retry — permanent until
  a manual pod restart.

The differentiator is not versioning or cache-key cleanup; it is simply whether that secret gets
pushed again during the window.

---

Fix

Restoring initial_fetch_timeout: 0 on generated SDS config sources addresses both halves: it
removes the byte change that orphans providers, and it restores fail-closed behaviour so a starved
provider leaves the listener warming instead of activating without certificates.

Verified independently: with 0s in place, pods with an unservable secret stay NotReady and are
excluded from Service endpoints, rather than going Ready and blackholing TLS.

---

Evidence status

• Verified by source and golden diff: the sds_config byte change between v1.9.0 and v1.9.1.
• Verified by Envoy source (v1.38.1, identical at HEAD): findAdditions suppression,
  findRemovals not re-requesting, SharedTargetImpl immediate-ready, the weak_ptr provider cache,
  and the fail-open timeout that leaves the watch registered.
• Verified experimentally (emulated): forcing exactly that sds_config transition on a warm
  proxy (0 restarts) produced 55/55 secrets timing out at +15.1s, listeners active with 0
  warming, readiness 200, ssl_connection_error 0, and per-secret recovery on rotation.
• Verified experimentally (real two-binary upgrade): a genuine
  envoyproxy/gateway:v1.9.0 → v1.9.1 controller image swap reproduced the same signature on
  7 never-restarted, non-terminating old-generation pods — 55 timeouts each, readiness 200,
  0/55 SNIs serving. See "Confirmed result of the two-binary run" above. No inference remains
  in the causal chain.

Pitfalls found while reproducing

• You cannot pin your way out of the proxy rollout. The bootstrap is embedded in the pod
  template and changes between versions. Use the gradual-rollout window instead.
• Terminating pods produce a false positive. A draining pod reports ready=200 with dead TLS
  because the readiness listener outlives the TLS listeners. Always filter on
  Running + no deletionTimestamp.
• kubectl port-forward on the TLS port dies the moment the listener is replaced
  (connection reset by peer), which reads as a total outage. Probe from an in-cluster pod against
  the pod IP instead.
• envoy_sds_update_attempt is not a wire-send signal — old and new providers for a secret
  share a stat scope keyed on the secret name, and it increments on six call sites. Use
  envoy_sds_update_success.

[arkodg]

cc @zirain @zhaohuabing