fix(api-gateway): populate route WatchSet before watching

[ian-dirac]

Description

Problem. API gateway route reconciliation can crash a Consul server with concurrent map iteration and map write when a route references multiple backend services.

Root cause. After reading the first backend discovery chain, reconciliation passes ws.WatchCtx to the asynchronous controller. The route loop then continues populating the same memdb.WatchSet for later backends while WatchCtx may already be iterating the WatchSet's underlying map.

Fix. Read and validate every backend discovery chain before registering the trigger. Once WatchCtx is handed to the controller, reconciliation no longer mutates its WatchSet. If state changes between a discovery-chain read and trigger registration, the existing watch channel is already closed, so the newly registered trigger returns immediately and enqueues another reconciliation.

The shared reconciliation path covers HTTP and TCP API gateway routes. This does not change any public API, configuration schema, route matching, or request-routing behavior.

Testing & Reproduction steps

Runtime reproduction shape: configure an API gateway route with at least two distinct backend service names, then update the route or related discovery-chain config entries to cause repeated reconciliation. The affected ordering is:

read backend A into ws -> start ws.WatchCtx -> read backend B into ws

The last step can mutate the WatchSet while the controller goroutine is iterating it. The panic is scheduler-dependent; #23522 includes reduced HCL and the full source trace.

Deterministic regression coverage:

• TestAPIGatewayControllerPopulatesWatchSetBeforeRegisteringTrigger requires both backend discovery chains to finish processing before AddTrigger. Against the previous ordering it fails at trigger registration (want 2, got 0); it passes with this change.
• TestDiscoveryChainController now verifies the missed-update edge case: a discovery-chain change made after the state read but before AddTrigger closes the existing watch channel, and registering that WatchSet immediately enqueues reconciliation.

Validation run:

• gofmt -s and goimports -local github.com/hashicorp/consul/
• golangci-lint v2.11.4 with hashicorpmetrics (repository-wide on amd64; changed packages on 386)
• CGO_ENABLED=0 go build -tags=hashicorpmetrics -o /dev/null .
• go test -tags=hashicorpmetrics ./agent/consul/gateways ./agent/consul/controller
• CGO_ENABLED=1 go test -tags=hashicorpmetrics -race -gcflags=all=-d=checkptr=0 ./agent/consul/gateways ./agent/consul/controller
• GOARCH=386 CGO_ENABLED=0 go test -tags=hashicorpmetrics ./agent/consul/gateways ./agent/consul/controller
• Both focused regression tests repeated 50 times

Links

Fixes #23522

PR Checklist

• updated test coverage
• external facing docs updated
• appropriate backport labels added
• not a security concern

PCI review checklist

• I have documented a clear reason for, and description of, the change I am making.

• If applicable, I've documented a plan to revert these changes if they require more than reverting the pull request.

• If applicable, I've documented the impact of any changes to security controls.

[hashicorp-cla-app]

Thank you for your submission! We require that all contributors sign our Contributor License Agreement ("CLA") before we can accept the contribution. Read and sign the agreement

Learn more about why HashiCorp requires a CLA and what the CLA includes

_{Have you signed the CLA already but the status is still pending? Recheck it.}