Ingress and TCP CR that share one Secret: TCP certificate marked unused and HAProxy reloaded on every sync

[mkutlak]

Summary

An Ingress and a TCP custom resource in one namespace use the same TLS Secret. The controller then marks the TCP copy of the certificate as unused on some syncs. The cleanup tries a runtime delete that cannot work for TCP binds, and forces a HAProxy reload. A later sync registers the certificate again. This repeats with no change to any Kubernetes object. On our clusters this causes up to 76 reloads per controller pod per day.

Environment

• The controller image is haproxytech/kubernetes-ingress:3.2.9, from Helm chart kubernetes-ingress 1.52.0.
• The controller runs with no --namespace-whitelist
• The related code is identical in v3.2.14 and on master (e904182).

What we see

The controller logs this line and then reloads HAProxy. No object in the cluster changed.

INFO    haproxy/certs/main.go:325 [transactionID=...] reload required : Runtime delete of cert file 'apps_app-tls.pem' failed : /var/run/haproxy-runtime-api.sock [3]  Can't delete the entry: crt-list '/etc/haproxy/certs/tcp' does not exist! [del ssl crt-list /etc/haproxy/certs/tcp /etc/haproxy/certs/tcp/apps_app-tls.pem] not found
[NOTICE]   (66) : Reloading HAProxy
INFO    controller/controller.go:229 [transactionID=...] HAProxy reloaded

In 24 hours one pod logged 76 reloads and 77 of these lines. Every reload had this cause. The intervals are irregular, from one minute to about 40 minutes.

Inside the pod:

• /etc/haproxy/certs/tcp/apps_app-tls.pem exists. Its mtime always equals the time of the last reload.
• The TCP frontend bind is crt /etc/haproxy/certs/tcp/apps_app-tls.pem ssl, a single file.
• show ssl crt-list lists only /etc/haproxy/certs/frontend. HAProxy creates a crt-list (a certificate list that the runtime API can edit) only for a directory given to crt. No crt-list exists for /etc/haproxy/certs/tcp.

Root cause

Three defects combine. Line numbers are for v3.2.14.

1. The secret manager records processed Secrets by namespace and name only, without the certificate type (secret.go#L56-L64). When the Ingress copy (FT_CERT) of a Secret was processed, Store() skips the TCP copy (TCP_CERT) of the same Secret. AddSecret never runs for the TCP directory, so the TCP entry keeps inUse = false.

2. The processed set leaks from the Ingress loop into the TCP CR handler. processIngressesDefaultImplementation resets SecretsProcessed at the start of each namespace (controller.go#L541). processIngressesWithMerge does the same (controller.go#L401). Nothing resets the set before the update handlers run (controller.go#L180). As a result, the set still holds the Secrets of the namespace that the loop visited last. Go iterates maps in random order. When the namespace of the TCP CR is last, Store() skips the TCP registration in applyBindOverride (tcp-cr.go#L234-L250). This is why it happens on about one sync in N, where N is the number of namespaces.

3. refreshCerts treats every certificate directory as a crt-list. deleteRuntime always calls del ssl crt-list <dir> <file> first (certs/main.go#L231-L239). TCP CR binds use crt <file>, so the call always fails for the TCP directory. The failure forces a reload (certs/main.go#L325) and refreshCerts removes the entry from its map. On a later sync the TCP handler registers the Secret again, because the entry is gone, and writes the file.

The cycle: sync A visits the TCP namespace last, marks the TCP certificate unused, fails the runtime delete, and reloads. Sync B registers the certificate again. Repeat.

Steps to reproduce

1. Run the controller with --ingress.class=haproxy and no namespace whitelist in a cluster with Ingress objects in several namespaces.
2. In one namespace, create a TLS Secret app-tls.
3. In that namespace, create an Ingress with spec.tls[0].secretName: app-tls.
4. In the same namespace, create the TCP CR like one below.
5. Watch the controller log for 30 minutes.

apiVersion: ingress.v3.haproxy.org/v3
kind: TCP
metadata:
  name: app-tcp
  namespace: apps
  annotations:
    ingress.class: haproxy
spec:
  - name: app
    frontend:
      name: app-tcp-frontend
      mode: tcp
      binds:
        app:
          name: app
          port: 5671
          ssl: true
          ssl_certificate: app-tls
    service:
      name: app-broker
      port: 5672

Expected: no reload, because no object changed.

Actual: the log line above appears at random intervals, and HAProxy reloads each time.

Impact

Each reload starts a new worker and keeps the old worker until its connections close or hard-stop-after fires. With long-lived TCP connections, every spurious reload later drops those connections. We saw three to four old workers stacked per pod at any time.

Workaround

• Set clean-certs: "false" in the controller ConfigMap. The cleanup pass no longer runs, and the reloads stop. Unused certificate files then stay on disk until the pod restarts.
• Or give the TCP CR a Secret with a different name than the Ingress.

Suggested fix

1. Key SecretsProcessed by namespace, name and SecretType, or keep one set per type.
2. Reset SecretsProcessed before the update handlers run, so that Ingress loop state does not reach the TCP CR handler.

[mkutlak]

Might be related: Issue 835