feat(memory): dest-connection age sweep + leak-2 probes - #84
Merged
Conversation
…ine projection - prepare_threshold=None + idle_in_transaction_session_timeout on the catalog connection (pgbouncer transaction-pooling safe) - in-transaction floor guard: refuse to plan below MIN(snapshot_id)-1 (closes the clamp's check-then-plan TOCTOU) - _read_inline projects per-store with additive NULL-fill; the missing+extra rename signature still raises (a boot-after-ADD-COLUMN reader previously wedged on UndefinedColumn against pre-ADD inline stores) - drop unused row_id_start from the plan SELECT
One process = one source table + one destination + one team filter, millpond posture: single loop, 3-attempts-then-crash, cursor strictly after commit, at-least-once with downstream uuid dedup. Reuses feed.py; the delete-assertion quartet (cursor-scoped) enforces the append-only contract operationally. Design + review history: per-destination-duckling.md.
- TestVariantFeed: pinned projection over a VARIANT-bearing table; read across a compaction merge of variant-bearing files - TestInline::test_variant_inline_plane: VARIANT in the inline store, mixed-plane read - duckling e2e: boot/deliver with a VARIANT companion; VARIANT added mid-stream (running process unaffected, restart re-excludes)
…tay out of the repo
Leak-2 (persistent_oom.md) is accumulation inside destination DuckDB instances, freed at connection close — hypothesis-2's Leak-B family. The source-connection recycle (C3) was falsified: RSS slope unchanged. This ships the working mitigation plus the instrumentation that found it. - memory.dest_conn_max_age_seconds (default 600, 0 disables, <60 rejected): DestinationPool.evict_older_than() closes aged pooled connections; budget-capped per poll cycle so a mass event can't storm the destination catalog; pinned entries skip (no mid-apply tear). - Config wiring + validation; viaduck_dest_conn_sweeps_total counter. - Probes (default-off, for the hunt): PYTHONTRACEMALLOC=25-compatible SIGUSR1 tracemalloc dump (flag-only handler; the poll loop does the work — no logging inside the signal handler); arrow pool gauges; the source-conn recycle probe (kept for the record of C3's falsify); VIADUCK_CDC_SPLIT_READ split-read workaround for the upstream ducklake_table_changes executor-stall wedge (wedge-table-changes.md). - destination.py create-table race retry: 'and' not 'or' — non-race errors raise immediately; exhaustion surfaces the real error, never a bare AssertionError. - Tests: sweep noop/disabled/aged-only/pinned-skip/budget-cap/eviction count; create-retry race-text + exhaustion; config validation for the new knob. 1066 tests pass; ruff clean.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
The dest-pool age sweep — the first intervention that measurably moves Leak-2 (
persistent_oom.md). Accumulation lives in destination DuckDB instances and is freed at connection close; closing pooled connections past an age is the mitigation.What's in it
memory.dest_conn_max_age_seconds(default 600)config.pyDestinationPool.evict_older_than()destination.pymain.pyviaduck_dest_conn_sweeps_totalmetrics.pymain.pyVIADUCK_TRACEMALLOC=1orPYTHONTRACEMALLOC=25metrics.pymain.pyVIADUCK_CDC_SPLIT_READsplit-readsource.pyducklake_table_changesexecutor stall (wedge-table-changes.md, issue forthcoming)create_table_if_not_existsretry fixdestination.pyor, retried every exception into a bare assert; nowand, real error surfacestests/unit/What it is not
self_recycle_enabledstays default-on; flipping it is a follow-up once the sweep proves out in prod.Validation