feat: destination lifecycle state machine (active|paused|draining|retired) - #56
Merged
Conversation
…ired) Per-destination operator intent in <state_table>_lifecycle (name derives from the cursor table: pipelines sharing a PG never share intent). Absent row = active; unknown value = paused + one ERROR per transition. States re-read every poll cycle — live pause with no restart; a state-store blip keeps last-known states (lifecycle shares the cursor store's PG, so fail-to-paused would turn any PG blip into a fleet-wide self-inflicted mass discard). Semantics (two adversarial review rounds + operator feedback): - paused/retired: controlled crash — buffer discarded via the FlushFail machinery (position rewound to the durable cursor, epoch bumped; lifecycle_discarded_rows_total), connection evicted only once CLEAN (an in-flight flush's retry loop re-creates the pool entry, so a transition-time evict leaks it for the stint); cursor = resume point. - a flush that SUCCEEDS after a discard restores position >= through in its success path (epoch-bumped) — without it, resume re-reads a committed range: deterministic duplicates in append_only. TLA-proven load-bearing (below). - draining: no new reads, flush out, evict when clean; the completion log distinguishes 'flushed out' from 'ended via a flush-failure rewind' (that range was NOT delivered; resume to re-read before retiring). - retired: terminal, never written by code (StateManager refuses; the ack is an operator UPDATE with updated_by). Viaduck SEVERS the cursor rows (all instances, idempotent per cycle + startup backstop) — re-add = new tenant = fresh seed per seed_mode, deterministic regardless of partition drift. - seeding is lifecycle-gated: paused destinations take no bulk writes and a broken one can't crashloop the instance; a skipped cursor-0 destination stays read-gated after resume until a restart seeds it. Observability: one-hot viaduck_destination_lifecycle_state, read-only /lifecycle (state + reason/updated_by/updated_at + staleness age), /status short-circuits to the lifecycle state (intentional pauses stop reading as 'lagging'; join lag alerts on state!="active"). README gains the operator runbook with the canonical SQL. TLA+: new PauseDest action (discard + rewind with the in-flight flush PRESERVED, unlike ProcessCrash/FlushFail) and the FlushCommit position restore. All 7 invariants hold over 85,012,333 distinct states (1.18B generated); removing the restore yields a 6-step BufferPositionBound counterexample (SrcInsert, BufferRead, FlushStart, PauseDest, FlushCommit) — the formal witness for the pause-races-in-flight-flush duplicate-delivery bug. M3 of the dynamic sourcing plan; the substrate C2 discovery drives (CP removal -> draining + alert; retire stays a human ack).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
M3 of the dynamic team-sourcing plan: per-destination operator-intent lifecycle states in a new
<state_table>_lifecyclePostgres table, re-read every poll cycle — live pause/drain/retire with no restart, for the legacy static destinations today and as the substrate the C2 discovery poller drives next (CP removal →draining+ alert;retiredstays a human ack, never automatic).activepauseddrainingretiredReview-hardened semantics
Two adversarial rounds (lead-QE + principal-SWE), all findings closed in-branch. Highlights:
position >= throughwith an epoch bump — otherwise resume re-reads a committed range (deterministic duplicates inappend_only). Pinned by a threaded race test and by the model checker (below).state.table): pipelines sharing a Postgres cannot pause each other's identically-named destinations.TLA+
New
PauseDestaction — discard + position rewind with the in-flight flush preserved (unlike ProcessCrash/FlushFail) — plus the FlushCommit position restore. All 7 invariants hold over 85,012,333 distinct states (1.18B generated, up from 19.9M pre-lifecycle). Deleting the restore produces a 6-stepBufferPositionBoundcounterexample (SrcInsert → BufferRead → FlushStart → PauseDest → FlushCommit) — the spec now permanently witnesses why that line of_flushexists.Operator surface
Documented SQL runbook in README (pause/resume/drain/retire), read-only
/lifecycle(state + reason/updated_by/updated_at + staleness age for keep-last-known), one-hotviaduck_destination_lifecycle_state{destination,state},viaduck_lifecycle_discarded_rows_total, and/statusshort-circuiting to the lifecycle state so intentional pauses stop reading as "lagging" (join lag alerts onstate!="active").Testing
just ciexit 0: 564 unit (20+ new: tracker semantics, discard/epoch/suspension, threaded discard-vs-inflight-flush race, /lifecycle payload) + 79 integration (lifecycle round-trips, per-pipeline isolation, DB CHECK, cross-instance cursor severing — real Postgres), fmt/lint/semgrep/docker green. TLC clean as above.