Gateway API Migration Guide

Overview & Rationale

UnderStack has migrated from the legacy Ingress API with ingress-nginx controller to the new Kubernetes Gateway API with Envoy Gateway. This migration was driven by the ingress-nginx project retirement announcement with a deadline of March 2026. The Kubernetes community recommends migrating to the Gateway API as the modern, standardized approach for managing ingress traffic.

Benefits of Gateway API

Role-oriented design: Separates concerns between cluster operators (Gateway) and application developers (Routes)
Expressive routing: More powerful matching and filtering capabilities
Protocol support: Native support for HTTP, HTTPS, TLS passthrough, TCP, and UDP
Extensibility: Vendor-specific features through policy attachments
Type safety: Strongly typed API with better validation

Architecture Overview

UnderStack uses a centralized Gateway architecture where cluster operators manage shared Gateway resources, and application teams create Route resources in their own namespaces.

graph TB
    subgraph "Cluster Infrastructure (envoy-gateway namespace)"
        EGC[Envoy Gateway Controller]
        GWE[Gateway: external]
        GWI[Gateway: internal]
        EGC --> GWE
        EGC --> GWI
    end

    subgraph "Application Namespaces"
        subgraph "openstack namespace"
            HR1[HTTPRoute: keystone]
            HR2[HTTPRoute: horizon]
            HR3[HTTPRoute: nova]
            SVC1[Service: keystone-api]
            SVC2[Service: horizon]
            SVC3[Service: nova-api]
            HR1 --> SVC1
            HR2 --> SVC2
            HR3 --> SVC3
        end

        subgraph "argo namespace"
            HR4[HTTPRoute: argo-workflows]
            SVC4[Service: argo-server]
            HR4 --> SVC4
        end
    end

    HR1 -.references.-> GWE
    HR2 -.references.-> GWE
    HR3 -.references.-> GWE
    HR4 -.references.-> GWE

    Internet((Internet)) --> GWE
    InternalTraffic((Internal)) --> GWI

Concept Mapping

Understanding how Ingress concepts map to Gateway API helps with migration and troubleshooting:

Ingress Concept	Gateway API Concept	Location	Description
IngressController (ingress-nginx)	Gateway Controller (Envoy Gateway)	`envoy-gateway` namespace	The controller implementation that manages traffic
IngressClass	GatewayClass	Cluster-scoped	Defines which controller handles the Gateway
Ingress	Gateway	`envoy-gateway` namespace	Defines listeners (ports, protocols, TLS) - shared cluster resource
Ingress rules	HTTPRoute / TLSRoute	Application namespace	Defines routing rules (hostnames, paths, backends)
`nginx.ingress.kubernetes.io/backend-protocol` annotation	Backend resource	Application namespace	For TLS backends, uses Envoy Gateway's Backend CRD
`cert-manager.io/cluster-issuer` annotation	Gateway annotation	On Gateway resource	Certificate management moved to Gateway level

Key Architectural Differences

Ingress (Old):

Each application creates its own Ingress resource
Ingress includes both infrastructure concerns (TLS, listeners) and routing rules
Annotations used heavily for controller-specific features
Each Ingress can specify its own TLS configuration

Gateway API (New):

Cluster operators create shared Gateway resources (external, internal)
Applications create lightweight Route resources that reference Gateways
Clear separation: Gateway = infrastructure, Route = application routing
TLS certificates managed at Gateway level for all routes
Routes can be in different namespaces than the Gateway

UnderStack Implementation

Gateway Resources

UnderStack deploys two shared Gateways in the envoy-gateway namespace:

External Gateway (gw-external):

Handles public-facing traffic
HTTPS (port 443) with TLS termination
Dynamically creates listeners based on configured routes
Integrates with cert-manager for automatic certificate provisioning
Each hostname gets its own listener with dedicated TLS certificate

Internal Gateway (gw-internal):

Handles cluster-internal traffic
HTTP (port 80) without TLS
Single listener accepting all namespaces

Route Configuration

Routes are managed through the envoy-configs Helm chart, which generates HTTPRoute and TLSRoute resources based on declarative configuration.

HTTP Routes

HTTP routes terminate TLS at the Gateway and forward HTTP traffic to backend services:

routes:
  http:
    - name: keystone
      fqdn: keystone.example.com
      namespace: openstack
      service:
        name: keystone-api
        port: 5000

This generates:

A listener on the external Gateway for keystone.example.com:443
An HTTPRoute in the openstack namespace
A cert-manager Certificate resource named keystone-tls

TLS Routes (Passthrough)

TLS routes pass encrypted traffic directly to the backend without termination:

routes:
  tls:
    - name: argo-workflows
      fqdn: workflows.example.com
      namespace: argo
      service:
        name: argo-server
        port: 2746

Used when the backend service handles TLS itself (like Argo Workflows with HTTPS).

HTTPS Backends

For backends that expect HTTPS but you want TLS termination at the Gateway:

routes:
  http:
    - name: argo-workflows
      fqdn: workflows.example.com
      namespace: argo
      service:
        name: argo-server
        port: 2746
        backendType: tls  # Tells Envoy to use HTTPS to backend

This uses Envoy Gateway's Backend CRD to establish TLS connections to the backend service.

Migration Guide

Prerequisites

Ensure your UnderStack deployment is using a recent version that includes:

Envoy Gateway component (automatically deployed via ArgoCD)
cert-manager with Gateway API support enabled
Updated bootstrap script with Gateway API configuration

Migration Steps for Existing Deployments

The migration requires changes in your deployment repository (deploy/<site>/). Here's the complete process:

1. Disable ingress-nginx Controller

In your apps.yaml, add ingress-nginx to the skip list:

- component: ingress-nginx
  skip: true

2. Create GatewayClass Resource

Create envoy-gateway/gateway-class.yaml:

---
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: eg
spec:
  controllerName: gateway.envoyproxy.io/gatewayclass-controller

And envoy-gateway/kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - gateway-class.yaml

3. Configure Gateway and Routes

Create envoy-configs/values.yaml with your Gateway and route definitions:

# yaml-language-server: $schema=https://rackerlabs.github.io/understack/schema/component-envoy-configs.schema.json
gateways:
  external:
    name: external-gateway
    namespace: envoy-gateway
    className: eg
    serviceAnnotations:
      # If using Cilium L2 announcements
      lbipam.cilium.io/ips: "10.4.35.199"
      lbipam.cilium.io/sharing-cross-namespace: "openstack"
      lbipam.cilium.io/sharing-key: "provisioning-service"
  internal:
    name: internal-gateway
    namespace: envoy-gateway
    className: eg
    serviceAnnotations:
      lbipam.cilium.io/ips: "10.4.35.199"
      lbipam.cilium.io/sharing-cross-namespace: "openstack"
      lbipam.cilium.io/sharing-key: "provisioning-service"

routes:
  http:
    # OpenStack services
    - fqdn: keystone.example.com
      service:
        name: keystone-api
        port: 5000
    - fqdn: nova.example.com
      service:
        name: nova-api
        port: 8774
    - fqdn: neutron.example.com
      service:
        name: neutron-server
        port: 9696
    # Add all your other services...

    # Services with HTTPS backends
    - fqdn: nautobot.example.com
      namespace: nautobot
      service:
        name: nautobot-default
        port: 443
        backendType: tls
    - fqdn: workflows.example.com
      namespace: argo
      service:
        name: argo-server
        port: 2746
        backendType: tls

Create empty envoy-configs/kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources: []

4. Remove Old Ingress Resources

Delete any standalone Ingress manifests:

# Example
rm argo-workflows/ingress.yaml

Update kustomization files to remove ingress references:

# argo-workflows/kustomization.yaml
resources:
  - https://github.com/rackerlabs/understack.git//components/argo-workflows/?ref=v0.0.43
  # Remove: - ingress.yaml

5. Disable Ingress in Helm Charts

For components that generate their own Ingress resources (like monitoring stack), disable them in their values:

# monitoring/values.yaml
grafana:
  ingress:
    enabled: false

prometheus:
  ingress:
    enabled: false

alertmanager:
  ingress:
    enabled: false

6. Update Cilium L2 Policies

If using Cilium L2 announcements, update the service selector to match Envoy Gateway:

# cilium/c-l2policies.yaml
spec:
  serviceSelector:
    matchLabels:
      app.kubernetes.io/name: envoy  # Changed from: app.kubernetes.io/instance: ingress-nginx

7. Update Load Balancer IP Sharing

Update any services that share IPs with the ingress controller:

# ironic/services-ironic.yaml
metadata:
  annotations:
    lbipam.cilium.io/sharing-cross-namespace: envoy-gateway  # Changed from: ingress-nginx

8. Update cert-manager Configuration

If not already done, ensure cert-manager has Gateway API support enabled in cert-manager/values.yaml:

config:
  enableGatewayApi: true
  apiVersion: "controller.config.cert-manager.io/v1alpha1"
  kind: "ControllerConfiguration"

9. Apply Changes

Commit and push your changes. ArgoCD will automatically:

Deploy Envoy Gateway controller
Create the GatewayClass
Create Gateway resources with listeners
Create HTTPRoute/TLSRoute resources
Provision TLS certificates via cert-manager
Remove the old ingress-nginx controller

Adding New Applications

When adding a new application that needs external access:

Add route configuration to your site's envoy-configs/values.yaml:

routes:
  http:
    - name: myapp
      fqdn: myapp.example.com
      namespace: myapp-namespace
      service:
        name: myapp-service
        port: 8080

For HTTPS backends, add backendType: tls:

routes:
  http:
    - name: myapp
      fqdn: myapp.example.com
      namespace: myapp-namespace
      service:
        name: myapp-service
        port: 8443
        backendType: tls

For TLS passthrough (no termination at Gateway):

routes:
  tls:
    - name: myapp
      fqdn: myapp.example.com
      namespace: myapp-namespace
      service:
        name: myapp-service
        port: 8443

Do not create Ingress resources - all routing is managed centrally through envoy-configs.yaml

For OpenStack Helm charts, ensure these settings are configured:

network:
  use_external_ingress_controller: true

manifests:
  ingress_api: false
  service_ingress_api: false

endpoints:
  identity:
    scheme:
      public: https
    port:
      api:
        public: 443

Troubleshooting

Check Gateway status:

kubectl get gateway -n envoy-gateway
kubectl describe gateway gw-external -n envoy-gateway

Check Route status:

kubectl get httproute -A
kubectl describe httproute <name> -n <namespace>

View Envoy Gateway logs:

kubectl logs -n envoy-gateway -l control-plane=envoy-gateway

Check generated Envoy configuration:

# Find the Envoy proxy pod
kubectl get pods -n envoy-gateway -l gateway.envoyproxy.io/owning-gateway-name=gw-external

# View configuration
kubectl exec -n envoy-gateway <pod-name> -- curl localhost:19000/config_dump

Common issues:

Route not working: Check that the HTTPRoute's parentRefs correctly references the Gateway name and namespace
Certificate issues: Verify the Gateway annotation cert-manager.io/cluster-issuer is set and the ClusterIssuer exists
Backend connection failures: For HTTPS backends, ensure backendType: tls is set and the Backend resource is created
No traffic reaching HTTPRoutes: Check if the Envoy Gateway Service has an external IP assigned:
```
kubectl get svc -n envoy-gateway
```
If the Service is stuck without an external IP, the old ingress-nginx Service may still exist and be holding the port. The skip: true setting only prevents new resources from being created but does not prune existing ones. Manually remove the old ingress-nginx resources:
```
# Check for old ingress-nginx resources
kubectl get svc -n ingress-nginx
kubectl get deploy -n ingress-nginx

# Remove the old controller
kubectl delete namespace ingress-nginx
```

Configuration Reference

envoy-configs Helm Chart

The envoy-configs chart is deployed via ArgoCD and generates Gateway and Route resources.

Location: components/envoy-configs/

Deployment: apps/site/envoy-configs.yaml

Site-specific values: deploy/<site>/envoy-configs/values.yaml

Gateway Schema

gateways:
  external:                     # or internal
    name: string                 # Required: name of the Gateway resource
    namespace: string            # Required: namespace where the Gateway is deployed
    className: string            # Required: GatewayClass name (e.g., "eg")
    serviceAnnotations: {}       # Optional: annotations for the generated Service
    externalTrafficPolicy: string # Optional: "Cluster" or "Local"
    envoyDeployment:             # Optional: tune the Envoy proxy pods for this Gateway
      replicas: integer          # Number of Envoy proxy pod replicas (minimum: 1)
      container:
        resources:
          requests:
            cpu: string
            memory: string
          limits:
            cpu: string
            memory: string

envoyDeployment is unset by default, which keeps Envoy Gateway's own defaults (a single replica with no resource requests or limits). Raise replicas and set container.resources when a Gateway is fronting a service that needs to absorb request bursts, for example:

gateways:
  external:
    name: external-gateway
    namespace: envoy-gateway
    className: eg
    envoyDeployment:
      replicas: 3
      container:
        resources:
          requests:
            cpu: "250m"
            memory: "256Mi"
          limits:
            cpu: "1"
            memory: "512Mi"

Scale the Gateway's Envoy proxies, not the backend service's own pods, when the bottleneck is connection handling or TLS termination in front of the service rather than the service itself.

Route Schema

routes:
  http:
    - name: string              # Route name (optional, defaults to first part of fqdn)
      fqdn: string              # Fully qualified domain name
      namespace: string         # Namespace for the HTTPRoute resource
      path: string              # Path prefix (default: "/")
      pathType: string          # PathPrefix or Exact (default: "PathPrefix")
      service:
        name: string            # Backend service name
        port: integer           # Backend service port
        backendType: string     # Optional: "tls" for HTTPS backends
      filters: []               # Optional: Gateway API filters
      from: string              # Optional: "All" or "Selector" for namespace selection
      selector: {}              # Optional: Label selector for allowed namespaces

  tls:
    - name: string              # Route name (optional)
      fqdn: string              # Fully qualified domain name
      namespace: string         # Namespace for the TLSRoute resource
      service:
        name: string            # Backend service name
        port: integer           # Backend service port

Navigation Menu

Repository navigation

gateway-api.md

gateway-api.md

Gateway API Migration Guide

Overview & Rationale

Benefits of Gateway API

Architecture Overview

Concept Mapping

Key Architectural Differences

UnderStack Implementation

Gateway Resources

Route Configuration

HTTP Routes

TLS Routes (Passthrough)

HTTPS Backends

Migration Guide

Prerequisites

Migration Steps for Existing Deployments

1. Disable ingress-nginx Controller

2. Create GatewayClass Resource

3. Configure Gateway and Routes

4. Remove Old Ingress Resources

5. Disable Ingress in Helm Charts

6. Update Cilium L2 Policies

7. Update Load Balancer IP Sharing

8. Update cert-manager Configuration

9. Apply Changes

Adding New Applications

Troubleshooting

Configuration Reference

envoy-configs Helm Chart

Gateway Schema

Route Schema

Additional Resources

FilesExpand file tree

gateway-api.md

Latest commit

History

gateway-api.md

File metadata and controls

Gateway API Migration Guide

Overview & Rationale

Benefits of Gateway API

Architecture Overview

Concept Mapping

Key Architectural Differences

UnderStack Implementation

Gateway Resources

Route Configuration

HTTP Routes

TLS Routes (Passthrough)

HTTPS Backends

Migration Guide

Prerequisites

Migration Steps for Existing Deployments

1. Disable ingress-nginx Controller

2. Create GatewayClass Resource

3. Configure Gateway and Routes

4. Remove Old Ingress Resources

5. Disable Ingress in Helm Charts

6. Update Cilium L2 Policies

7. Update Load Balancer IP Sharing

8. Update cert-manager Configuration

9. Apply Changes

Adding New Applications

Troubleshooting

Configuration Reference

envoy-configs Helm Chart

Gateway Schema

Route Schema

Additional Resources