Backend default-local-service duplicates backend snippets

[hedgieinsocks]

Add the following to helm values:

controller:
  config:
    backend-config-snippet: |
      filter spoe engine coraza config /etc/haproxy/coraza/coraza.conf
      http-request deny if { var(txn.coraza.action) -m str deny }

See that haproxy fails to reload, inspect the faulty config in /etc/haproxy/failed and see that the default backend is indeed broken by duplication:

backend app_svc_default-local-service_http from haproxytech
  mode http
  balance roundrobin
  option forwardfor
  no option abortonclose
  default-server check
  ###_config-snippet_### BEGIN
  ### configmap:app/app-haproxytech-ingress-alpha ###
  filter spoe engine coraza config /etc/haproxy/coraza/coraza.conf
  http-request deny if { var(txn.coraza.action) -m str deny }
  ### service:app_svc_default-local-service_http/app/default-local-service ###
  filter spoe engine coraza config /etc/haproxy/coraza/coraza.conf
  http-request deny if { var(txn.coraza.action) -m str deny }
  ###_config-snippet_### END
  server SRV_1 127.0.0.1:6061 enabled

It's probably related to this part

kubernetes-ingress/pkg/controller/global.go

Lines 216 to 218 in fc12067

	} else {
	defaultLocalService.Annotations = k8sStore.ConfigMaps.Main.Annotations
	}

My chart is haproxytech/kubernetes-ingress 1.52.1

[hedgieinsocks]

I've removed these lines and it fixed the issue. But I'm not yet sure if this can backfire in other cases or situations.

[ivanmatmati]

Hi @hedgieinsocks , can you indicate which version of Helm Chart you're using ?

[hedgieinsocks]

@ivanmatmati sure, my chart is haproxytech/kubernetes-ingress 1.52.1

[ivanmatmati]

I'm puzzled because using the same 1.52.1 chart with values.yaml:

controller:
  config:
    backend-config-snippet: |
      http-send-name-header x-dst-server
      stick-table type string len 32 size 100k expire 30m
      stick on req.cook(sessionid)

I get a correct:

backend test_svc_default-local-service_http from haproxytech
  mode http
  balance roundrobin
  option forwardfor
  no option abortonclose
  default-server check
  ###_config-snippet_### BEGIN
  ### configmap:test/release-name-kubernetes-ingress ###
  http-send-name-header x-dst-server
  stick-table type string len 32 size 100k expire 30m
  stick on req.cook(sessionid)
  ###_config-snippet_### END
  server SRV_1 127.0.0.1:6061 enabled
  server SRV_2 127.0.0.1:1 disabled
  server SRV_3 127.0.0.1:1 disabled
  server SRV_4 127.0.0.1:1 disabled
  server SRV_5 127.0.0.1:1 disabled

Do you get systematically the same error ? Can you check the annotations on the service default-local-service ?

[hedgieinsocks]

Yes, I do get this systematically, and I only noticed it when I tried to configure Coraza because HAProxy started complaining that 2 identical filter spoe directives cannot live in one backend.

controller:
  config:
    backend-config-snippet: |
      http-send-name-header x-dst-server

backend foo_svc_default-local-service_http from haproxytech
  mode http
  balance roundrobin
  option forwardfor
  no option abortonclose
  default-server check
  ###_config-snippet_### BEGIN
  ### configmap:foo/foo-haproxytech-ingress-alpha ###
  http-send-name-header x-dst-server
  ### service:foo_svc_default-local-service_http/foo/default-local-service ###
  http-send-name-header x-dst-server
  ###_config-snippet_### END
  server SRV_1 127.0.0.1:6061 enabled

Perhaps it's surfaced only on bootstrap or something. Maybe you can try to kill your haproxy pods and see if fresh ones have the buggy config.

As for annotations, my understanding is that default-local-service is a virtual service that exists only in HAProxy world.

To confirm the image just in case, it is 3.2.12

[ivanmatmati]

Ok, I did the test with brand new deployment. Concerning the service, just ensure there's no existing service that could interfer with the default one (one with the location as indicated in the snippet comment).

[hedgieinsocks]

@ivanmatmati sorry for the delay, here is a quick, reliable reproduction of the issue with a fresh kind cluster:

kind create cluster
helm repo add haproxytech https://haproxytech.github.io/helm-charts
helm repo update
helm install my-release haproxytech/kubernetes-ingress -f values.yaml
### values.yaml
# controller:
#   config:
#     backend-config-snippet: |
#     http-send-name-header x-dst-server
keti deploy/my-release-kubernetes-ingress -- cat /etc/haproxy/haproxy.cfg | sed -n '/BEGIN/,/END/p'
  ###_config-snippet_### BEGIN
  ### configmap:default/my-release-kubernetes-ingress ###
  http-send-name-header x-dst-server
  ### service:default_svc_default-local-service_http/default/default-local-service ###
  http-send-name-header x-dst-server
  ###_config-snippet_### END