Repository navigation
Commit b56243c
authored
🤖 feat: hold CoderWorkspacePool members at a lowered owner cap, and keep the failure run across a rescale (#379)
## Summary
This PR closes Codex security finding 4271920249 from #359, and it keeps
the failure run of a `CoderWorkspacePool` across a rescale. Both changes
are in the dormant pool and member reconcilers. Nothing registers them
yet, so no user sees a change.
1. **A lowered `maxWorkspaces` holds.** When an admin lowers the owner's
`maxWorkspaces`, the pool can already have members that it created under
the old cap and that have not sent their create request yet. Before this
PR, those members still created workspaces above the new cap. Now:
- The member checks the cap again right before it writes its create
marker, after all its Coder reads, on a new uncached read of the control
plane. It lists the namespace's members without the cache and counts the
owner's other members on the same control plane that may have a
workspace (a create marker or a workspace ID) and whose workspace is not
deleted. At the cap, it waits with `OwnerWorkspaceLimit` and sends no
create request.
- The pool deletes its own members that sent no create request yet,
newest first, while the owner has more live members than the cap allows.
Otherwise these members would wait at their cap check until their
deadline fails them, and each such failure would count toward the
failure limit. Each delete carries the listed UID and `resourceVersion`,
so a member that wrote its marker after the list answers 409 and stays.
2. **A rescale keeps the failure run.** Before this PR, any spec change
(`metadata.generation`) started a new run of failures. So a pool that an
HPA rescales through `/scale` could get past the 5-failure limit. Now
only a change of `status.resolved` starts a new run: the template, the
version, the preset, the parameters, the owner, or
`readyTimeoutSeconds`. A change of `replicas` or `maxInFlight` keeps the
run. `maxWorkspaces` still caps creation either way.
Refs #153
## Background
This is PR 10a of the `CoderWorkspacePool` stack (#153). PR 10b (delete
order, new before old, used-member release, consent removal) stacks on
this branch.
Decision on the run reset: the maintainer asked that a replicas-only
change does not reset the run, and that only input changes that can fix
a failure do. I added `readyTimeoutSeconds` to the list, because a
longer timeout can fix a `DeadlineExceeded` failure. Without it, a pool
that stalled on deadline failures could only leave `MemberFailureLimit`
through a template, version, or parameter change. To detect that change
without a stored copy of the spec, `status.resolved` now holds
`readyTimeoutSeconds`, and new members pin it from there. The member
input check does not compare it, so a timeout change does not make ready
members outdated. The first resolve (no stored `status.resolved`) does
not start a run.
This also removes the case where the failure write before the deletes
stores the run but not `observedGeneration`: the reset no longer reads
`observedGeneration`. I recorded the remaining effect (that write
publishes an old `observedGeneration` until the final write) in the plan
for PR 12, as asked.
API: one new optional status field,
`status.resolved.readyTimeoutSeconds`. The CRD is still in
`config/crd/dormant/`, and the pool has no API reference page yet, so
`make docs-reference` changes nothing.
## Validation
Red, then green. I wrote the two tests first and ran them on `main`
(`fb42486a`):
```text
--- FAIL: TestPoolFailureRunSurvivesRescale
coderworkspacepool_failure_test.go:223: a rescale keeps the run (expected: 1, actual: 0)
--- FAIL: TestPoolLoweredOwnerCap
coderworkspacepool_cap_test.go:44: one workspace at maxWorkspaces 1 (expected: 1, actual: 3)
```
With this PR, both pass.
- `TestPoolLoweredOwnerCap` (new, `coderworkspacepool_cap_test.go`): 3
pending members, then `maxWorkspaces` goes from 10 to 1. Every member
reconciles before the pool sees the new cap: Coder gets exactly 1
workspace, and the other two wait with `OwnerWorkspaceLimit` without a
marker. The pool then deletes exactly those two, keeps the member with
the workspace, and creates no new member (`OwnerWorkspaceLimit`).
- `TestPoolFailureRunSurvivesRescale` (new): after one failure, a
`/scale` write to 2 replicas, a `maxInFlight` change, and a new name of
the same version keep `consecutiveFailures: 1` and the backoff. A
`readyTimeoutSeconds` change starts a new run, a failure of a member
with the old timeout does not count in it, and the new members pin the
new timeout.
- Changed tests: `TestPoolFailureBackoffAndLimit` and
`TestPoolFailureRunResets` now lift the limit with a
`readyTimeoutSeconds` change instead of a `maxInFlight` change.
`TestPoolResolvesInputs` expects the new field.
Codex review round 1 (all fixed in `ee015d1b`):
- 4274233487: a deleting member that sent no create request no longer
counts as live, so the next reconcile deletes no more members
(`TestPoolLoweredOwnerCapDeletesOnce`).
- 4274233497: a new name of the same version no longer starts a new run
(`poolRunInputsChanged` ignores `templateVersionName`).
- 4274233505: a failed member with an old `readyTimeoutSeconds` no
longer counts in the new run. It is still deleted.
Codex review round 2 (fixed in `007545e9`):
- 4274565367: the member's cap check now runs last, after all Coder
reads, on a new uncached control plane read (deletion, consent, and
cap). `TestPoolMemberRechecksCapLast` answers the first control plane
read with the old cap and failed on the previous code.
- For 4274557698 on #380: the pool now reads the control plane without
the cache in every reconcile that resolved its inputs (`freshOwnerCap`),
before the cap trim and the creates, whatever the demand. Before, a pool
that needed no create skipped that read.
Codex review round 3 (fixed in `4612ad65`):
- 4274801753: while the pool's inputs do not resolve (any wait or stall
of the resolve step), failed members are deleted without counting. A bad
spec edit then leaves no failures in the run, and a revert to the same
inputs needs no reset (`TestPoolFailuresUncountedWhileUnresolved`).
Trade-off: failures during a Coder outage do not count either.
- 4274801768: the `MemberFailureLimit` message names only inputs that a
user can change (`spec.template` is immutable).
Mutation checks: I removed one control at a time, and each removal made
a named test fail (10 of 10): the member's cap check, the pool's deletes
above the cap, keeping members with a workspace out of those deletes,
counting only members that may have a workspace in the member's check,
the reset on a generation change, the resolved timeout, not counting
deleting pending members as live, ignoring a version rename, not
counting failures at an old timeout, and not counting failures while the
inputs do not resolve. The final cap read has a red run instead:
`TestPoolMemberRechecksCapLast` failed on the code before that fix.
Gates on the final commit: drift (codegen, manifests, installer, API
reference: no diff), `make docs-check`, dormant-CRD guard, docs lint and
spelling, `make verify-vendor`, `make build`, `make lint`, actionlint,
`make test`, `make test-integration`, `make vuln`, race. All passed.
## Size
468 hand-written lines (230 production, 238 test, counting removals).
Generated: 15 lines of the dormant CRD.
## Risks
None for users. The reconcilers are dormant. One limit, recorded in the
plan for PR 12: the member's cap check is exact while member reconciles
run one at a time (the controller-runtime default). With parallel member
reconciles, two members can pass the check together.
---
_Generated with `mux` • Model: `anthropic:claude-opus-5-5` • Thinking:
`high` • Cost: `$154.26`_
<!-- mux-attribution: model=anthropic:claude-opus-5-5 thinking=high
costs=154.26 -->
---------
Signed-off-by: Thomas Kosiewski <tk@coder.com>1 parent d56051f commit b56243c
9 files changed
Lines changed: 420 additions & 63 deletions
File tree
- api/v1alpha1
- config/crd/dormant
- internal/controller
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
232 | 232 | | |
233 | 233 | | |
234 | 234 | | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
235 | 239 | | |
236 | 240 | | |
237 | 241 | | |
| |||
293 | 297 | | |
294 | 298 | | |
295 | 299 | | |
296 | | - | |
297 | | - | |
| 300 | + | |
| 301 | + | |
| 302 | + | |
298 | 303 | | |
299 | 304 | | |
300 | 305 | | |
301 | 306 | | |
302 | 307 | | |
303 | 308 | | |
304 | | - | |
305 | | - | |
| 309 | + | |
| 310 | + | |
306 | 311 | | |
307 | 312 | | |
308 | 313 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
307 | 307 | | |
308 | 308 | | |
309 | 309 | | |
310 | | - | |
311 | | - | |
| 310 | + | |
| 311 | + | |
| 312 | + | |
312 | 313 | | |
313 | 314 | | |
314 | 315 | | |
315 | 316 | | |
316 | 317 | | |
317 | | - | |
318 | | - | |
| 318 | + | |
| 319 | + | |
319 | 320 | | |
320 | 321 | | |
321 | 322 | | |
| |||
386 | 387 | | |
387 | 388 | | |
388 | 389 | | |
| 390 | + | |
| 391 | + | |
| 392 | + | |
| 393 | + | |
| 394 | + | |
| 395 | + | |
389 | 396 | | |
390 | 397 | | |
391 | 398 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
0 commit comments