You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit aab11e7
Browse filesBrowse the repository at this point in the historyBrowse files
🤖 feat: write a cleanup record before each CoderAIGatewayKey create (#353 part 2c) (#385)
Refs #353 (part 2c: the `CoderAIGatewayKey` half of option B′, records).
Refs #156.
## Summary
Before each AI Gateway key create, the operator now writes a cleanup
record for the `CoderAIGatewayKey` object: a ConfigMap
`ck8s-cleanup-<object UID>` in the operator namespace, the same kind of
record that #374 added for `CoderAIProvider`. If the record cannot be
written, no key is created. The delete path removes the record only
after a LIST in the record's Coder deployment shows no key of the
object. In every other release without a delete (for example a namespace
deletion), the record stays. The sweep that uses it follows in the
stacked PR (part 2d).
## Design note (option B′, maintainer decision
#353 (comment))
How each invariant maps to keys:
1. **Record before create.** `issueKey` writes the record after the
Secret claim and dry run, and before the create marker (`pendingKey`,
`nextSerial`). A failed deployment ID read is a `CoderUnavailable` wait.
A failed write is an error. Either way no POST is sent and no marker is
written.
2. **No secrets.** Kind, UID, namespace, name, deployment ID, the key
name prefix `ck8s-<UID without dashes>-` (in the `nonce` field), the
time of the last create, and the current key ID. The test helper
`reconcileErr` now also checks the record for every key plaintext, in
every key test.
3. **Deployment ID from Coder** (`/api/v2/buildinfo`), read in the
reconcile that sends the create. When a record already names another
deployment, the create step stalls with `CoderDeploymentChanged` and
creates nothing in the new deployment (same rule as providers).
4.–6. Sweep rules: next PR.
7. **The record goes only after Coder confirms.** It goes at the end of
a normal delete (a LIST shows no key of the object). It also goes after
the one bounded pass, but only when the last create was more than 5
minutes (the settle window) before it, so no create can still commit
after that LIST. Before any delete pass, the operator checks that the
control plane still reaches the recorded deployment. If it does not, the
finalizer goes without a delete and the record stays. An object with
`nextSerial == 0` sent no create, so its record (left by a failed marker
write) goes with no Coder call.
8.–9. Same Role and namespace as #374. No RBAC change.
Key-specific choices:
- **One record per object, many keys.** A key object creates a new key
at every rotation and drift re-issue. The prefix finds all of them, so
the record stores only the current key ID, for admins. The ID is written
right after a successful issue, with no extra Coder call. If that write
fails, the next reconcile that sees the key listed writes it
(`recordAIGatewayKeyID`). The same path also gives a record to an object
from before records existed.
- **The record's create time is the marker's instant.** One
`attemptStart` stamps the record, `lastCreateAttemptTime` and the Secret
write-age limit, so the sweep's settle window covers every create.
Also in this PR: **#384 follow-up 4 for keys**
(#353 (comment),
item 4). A delete pass under the control plane hold now ends at the hold
end (`context.WithDeadline`), and its retry counts from the end of the
pass, as #384 did for `CoderMCPServer`.
## Validation
- New tests: `TestCoderAIGatewayKeyCleanupRecord`,
`TestCoderAIGatewayKeyNoCreateWithoutRecord`,
`TestCoderAIGatewayKeyDeploymentChanged`,
`TestCoderAIGatewayKeyRecordAfterBoundedPass`,
`TestCoderAIGatewayKeyRecordWithoutCreate`,
`TestCoderAIGatewayKeyRecordBackfill`, and
`TestAIGatewayKeyDeleteUnderHoldEndsAtHoldEnd`.
- Red then green: 11 mutations, each caught by a test: no record before
the create, no deployment-change check, no deployment check on delete,
the bounded pass always or never removing the record, a serial-0 delete
keeping the record, a release keeping the record, no issued ID, no
backfill, no hold deadline, and the requeue counted from the start of
the pass.
- Local gates: build-installer, codegen, docs-reference, drift,
verify-vendor, build, lint, govulncheck, test, test-integration, race
(`./internal/controller/...`), docs-check, actionlint. All passed.
- Kind dogfood with a licensed Coder runs on the stacked PR (part 2d),
because the sweep is needed for the namespace deletion scenario.
## Risks
- The key create path sends one more Coder request (`GET
/api/v2/buildinfo`) per create. A failure of that read blocks the create
with a 60 s retry, as for providers.
- Each reconcile with a listed current key does one live ConfigMap GET
in the operator namespace. Writes happen only when the ID changed.
---
_Generated with [`xum`](https://github.com/coder/xum) • Model:
`anthropic:claude-opus-5-5` • Thinking: `high`_
Signed-off-by: Thomas Kosiewski <tk@coder.com>
Copy file name to clipboardExpand all lines: docs/how-to/manage-ai-gateway-keys.md
+9-1Lines changed: 9 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -137,14 +137,22 @@ The finalizer stays while Coder cannot confirm the deletion:
137
137
|`ControlPlaneNotReady`, `OperatorAccessNotReady`| Every 60 seconds |
138
138
|`LicenseRequired`, `CoderRejected`, `NotSupported` (`Stalled=True`) | Every 10 minutes |
139
139
140
-
When the control plane is gone, nothing can delete the keys. The finalizer goes at once. The operator logs `Releasing the finalizer without deleting the keys in Coder` with reason `ControlPlaneGone` and the key IDs: delete those keys with `coder ai-gateway keys delete`.
140
+
When the control plane is gone, nothing can delete the keys. The finalizer goes at once, and the [cleanup record](#cleanup-records) stays. The operator logs `Releasing the finalizer without deleting the keys in Coder` with reason `ControlPlaneGone` and the key IDs: delete those keys with `coder ai-gateway keys delete`.
141
141
142
142
When you delete the control plane while `CoderAIGatewayKey` or `CoderAIProvider` objects that use it are being deleted, the control plane delete can take up to about 15 minutes. The control plane waits for the objects for up to 10 minutes: it keeps its finalizer, so Coder keeps running, and the objects delete their keys and providers as usual, with retries. The operator log shows `Waiting for the objects that use this control plane` with their names. The operator removes the workspace RBAC of the control plane before it waits, so Coder loses its Kubernetes access at once. An object that you do not delete holds the control plane for only 30 seconds, so delete the objects together with the control plane (or within 30 seconds). After 10 minutes, the objects that remain get at most 5 more minutes (30 seconds plus 10 seconds per object of one kind) for their bounded pass below, and then the control plane goes anyway. A foreground deletion (`kubectl delete --cascade=foreground`) does not wait, because Kubernetes then deletes the Coder Deployment first.
143
143
144
144
When the control plane does not wait (its namespace is being deleted, it is deleted in the foreground, or the 10 minutes have passed) and its operator access still works, the operator runs one pass of at most 10 seconds without the settle window, then removes the finalizer whatever the answer. A key create that was still in flight at that moment can commit later, and its key stays in Coder. When the pass fails, the log lists the IDs that it did not delete.
145
145
146
146
Each release without a delete counts in the controller metric `coder_k8s_cleanup_released_without_delete_total` (label `kind`), next to the log line.
147
147
148
+
### Cleanup records
149
+
150
+
Before each key create, the operator writes a cleanup record: a ConfigMap named `ck8s-cleanup-<object UID>` with the label `coder.com/cleanup-record=CoderAIGatewayKey` in the operator's namespace (`coder-system`). If it cannot write the record, it does not create the key. The record holds no key: it holds the object's namespace, name and UID, Coder's deployment ID from `/api/v2/buildinfo`, the key name prefix `ck8s-<UID without dashes>-` (the `nonce` field), the time of the last create, and the current key ID once Coder answered it. The deployment ID lives in the Coder database, so it names the database that holds the keys.
151
+
152
+
When the control plane later reaches another Coder deployment (for example after its database Secret changed), the record still locates the keys in the old one. The operator then stalls the object with `CoderDeploymentChanged` at its next key create, keeps the record and creates no key in the new deployment. A delete of the object then removes the finalizer without a delete in Coder. Delete the keys in the old deployment, then delete the ConfigMap.
153
+
154
+
The operator removes the record only after a list in the record's deployment shows no key of the object: at the end of a normal delete, or after the one bounded pass when the last create was more than 5 minutes before it. In every other release without a delete, the record stays and names where the keys are. Find the records with `kubectl -n coder-system get configmaps -l coder.com/cleanup-record=CoderAIGatewayKey`. The operator gets write access to ConfigMaps only in that namespace. Do not give other users write access to ConfigMaps there.
155
+
148
156
!!! warning "Delete the keys before the namespace"
149
157
When you delete the namespace, Kubernetes deletes the control plane and the `CoderAIGatewayKey` objects at the same time. The control plane is usually gone before the operator handles the objects, so the finalizers go without a delete, and the keys stay valid in Coder. The namespace deletion finishes, and the operator logs the key IDs with reason `ControlPlaneGone`.
returnerrors.New("assertion failed: AI Gateway key reconciler needs a manager, a client, an API reader, a scheme, a clock, and the operator namespace")
return ctrl.Result{}, errors.New("assertion failed: AI Gateway key reconciler needs a client, an API reader, a scheme, a clock, and the operator namespace")
144
146
}
145
147
// Only a wait after HTTP 429 keeps the backoff count.
returnr.releaseWithoutDelete(ctx, key, "the deployment check while CoderControlPlane is being deleted failed: "+other+aiGatewayKeyStepMessage(wait)), nil, nil
0 commit comments