controlplane: resolve workload issuers in the background instead of failing to start

[aojea]

Problem

Server.Start runs OpenID discovery for every issuer and fails after five attempts (#228's bounded retry), which agentmesh-one turns into FATAL Failed to start. Observed deploying agentmesh-one v0.1.0 to Cloud Run as a hub for an OpenShell cluster whose SPIRE OIDC provider did not exist yet:

OIDC discovery attempt 5/5 for http://35.253.119.53 failed: Get ".../.well-known/openid-configuration": context deadline exceeded
FATAL Failed to start: failed to start control plane: failed OIDC discovery: failed to create provider for http://35.253.119.53

Consequences: the hub cannot be deployed before the first platform's issuer exists; adding a platform is a redeploy; a workload issuer that goes away takes the hub down on its next restart, and with it enrollment, policy sync and /sts/token for every other member.

What this keeps and what it changes

The history matters here. Fail-fast dates from the hub decoupling (06fccab9); #228 added the bounded retry for rollouts; #266 fixed the console for the opposite failure, a one-shot discovery that failed silently and hid SSO for the pod's lifetime. The constraint from #266 is "an unresolved issuer must be loud and fail closed", not "must be fatal".

• --issuer (login, operator-fixed): unchanged, still fatal after the bounded retry. A typo should stop the process where it is seen.
• --workload-issuer entries: one bounded attempt at start-up; on failure a background resolver with exponential backoff (5s → 5m) keeps trying until the server stops. Until it resolves:
  • tokens from that issuer are refused with 401 and JWT validation failed: workload issuer <iss> is configured but its discovery document has not been resolved yet (on /register, /refresh, /token/exchange); a new identity.UnknownIssuerError lets the handlers tell that apart from an issuer that was never configured;
  • agentmesh_control_plane_workload_issuer_ready{issuer} is 0 (1 once resolved);
  • every retry is logged at WARN with the issuer, attempt and next delay; resolution at INFO.

The two discovery retry constants become package variables so tests can shorten them. Reference docs updated for both binaries.

Tests

• TestWorkloadIssuerResolvedInBackground: workload issuer answering 503 at start-up → Start succeeds, gauge 0, /token/exchange with a token from it → 401 naming it; issuer comes up → provider resolved without a restart, gauge 1, exchange 200.
• TestLoginIssuerDiscoveryFailureIsFatal: unreachable --issuer → Start returns an error naming it.
• go test ./internal/controlplane/ ./internal/identity/ -race clean; the two new tests clean over 338 runs under stress -p 8 with the race detector; ./hack/lint.sh 0 issues; no go.mod change (the gauge is read in tests through client_model, already a dependency).

Follow-up, not in this PR: deploying this build to the Cloud Run hub to confirm the start-before-issuer order end to end.

[gemini-code-assist]

Code Review

This pull request implements background resolution for workload OIDC issuers, ensuring that an unreachable workload issuer at startup does not prevent the control plane from starting. Instead, the control plane retries resolution in the background with exponential backoff, reports the status via a new Prometheus gauge, and returns a clear error message for tokens from unresolved issuers. In contrast, failures to resolve required login issuers remain fatal. The feedback highlights critical resource management issues in the new background resolution logic, specifically a potential TCP connection leak from cloning http.DefaultTransport without closing idle connections, a timer leak from using time.After in a loop, and a misleading log message regarding the next retry delay.

[aojea]

Verified on the Cloud Run hub with this branch's build, in the order that failed on v0.1.0:

1. SPIRE's OIDC discovery provider scaled to 0 (issuer http://35.253.119.53 unreachable), new hub revision deployed: {"status":"ready"}, log Workload issuer not resolved; tokens from it are refused until it answers, policy seeded, GKE nodes attached.
2. A sandbox call while unresolved: Border Crossing {"endpoint": "/token/exchange", "decision": "deny", "reason": "JWT validation failed: workload issuer http://35.253.119.53 is configured but its discovery document has not been resolved yet"}.
3. Provider scaled back to 1 at 07:12:38; Workload issuer resolved; tokens from it are accepted {"attempts": 5} at 07:13:29 on the same revision (backoff 5s → 10s → 20s → 40s).
4. The same sandbox call: mcp://calc/add(2, 3) = 5.0, mcp://payroll refused by policy.

[aojea]

On "do we keep rediscovering issuers / leak goroutines", from the code paths:

• Discovery runs once per process. Start → Init → discoverProviders; Init is only called from Start (standalone.go, control-plane/main.go). There is no periodic rediscovery before or after this PR; the issuer set is fixed by flags, so nothing is added or deleted at runtime and providers grows at most to the number of configured issuers.
• No duplicates within a pass. The seen map dedups across --issuer, --workload-issuer and <issuer>=<suffix>; an issuer in both lists stays in the required (fail-fast) set, as today.
• Goroutine bound and lifetime. One resolver per unresolved workload issuer, wg.Add/Done, exits on success or s.ctx.Done(); Close() cancels then wg.Wait(), and the in-flight discovery request is bound to s.ctx too.
• Gap closed in 44e96a0: Init is exported for embedders, so a second discovery pass was possible; it now skips issuers already resolved or in flight (resolving set under the providers mutex, cleared on success/shutdown). The test runs discovery twice and asserts one resolver, then none after resolution. Package clean under -race; 265 stress runs, 0 failures.