fix(gtc): prevent CDC data loss on poison records, streamed transactions, and reconnects

[emoss08]

Description

Fixes three silent data-loss / availability bugs in the GTC CDC pipeline:

1. Dead-letter queue now lets the pipeline continue. Previously, after exhausting retries and successfully writing a poison record to the DLQ, writeProjection still returned the error, which propagated up and killed the entire runtime — one permanently-bad record halted all replication. The retry loop is now split from the dead-letter decision: a record successfully parked in the DLQ is skipped (logged at error level and counted in a new gtc_dead_lettered_records_total metric) and the checkpoint advances past it. The runtime still fails hard when no DLQ is configured, when the DLQ write itself fails, or during shutdown (context cancellation never dead-letters). replay-dlq uses the raw write path so a failed replay surfaces to the operator instead of silently re-parking the entry and reporting success.
2. In-progress transaction streaming disabled. The reader requested streaming 'true' but the decoder never handled StreamCommit/StreamAbort, so any large transaction Postgres streamed before commit was silently dropped (and on abort, its records leaked into the next transaction). Streaming is now off (streaming 'false'): the consumer buffers all records until commit anyway, so streaming provided no consumer-side benefit, and pgoutput protocol v2 doesn't carry per-change subtransaction info, making correct subtransaction-abort handling impossible at this protocol version. The decoder now rejects stream messages with a loud error if the option is ever re-enabled, instead of corrupting silently.
3. Reconnects resume from the advanced LSN. The reconnect loop reused the LSN captured once at Start(), so any reconnect after hours of streaming replayed everything from the original position. It now resumes from the advanced client LSN (falling back to the start LSN only on first connect), and decoder state is reset on each new replication session so a half-received transaction from a broken connection can't bleed into the resumed stream.

Also wires the previously declared-but-never-recorded SinkErrors and RetryAttempts metrics in the sink retry loop.

Related Issue or Discussion

Found during a codebase completeness review; maintainer-driven session.

Type of Change

• Bug fix
• Feature
• Documentation
• Refactor
• Tests
• Build, CI, or infrastructure

Scope

• services/gtc/internal/core/services/runtime.go (+ tests)
• services/gtc/internal/adapters/primary/wal/decoder.go, reader.go (+ tests)
• services/gtc/internal/infrastructure/metrics/metrics.go

Validation

• Other: cd services/gtc && go build ./... && go vet ./... && go test ./... — all packages pass; gofmt -l clean on touched packages. golangci-lint could not run in this environment (binary built with Go 1.25, repo targets Go 1.26).
• cd services/tms && task test — not run; no TMS changes.
• cd services/tms && task lint — not run; no TMS changes.
• cd client && pnpm build — not run; no client changes.
• cd client && pnpm lint — not run; no client changes.

New test coverage: poison record → DLQ + checkpoint advance; halt when no DLQ configured; halt when DLQ write fails; replay surfaces sink failures without re-parking; decoder rejects stream messages; decoder reset clears transaction state; reconnect resume-LSN selection.

Deployment Notes

• Behavior change: a sink that stays down no longer halts the pipeline when a DLQ is configured — records park in the DLQ stream while the checkpoint advances. Watch gtc_dead_lettered_records_total and sink health; replay with replay-dlq once the sink recovers.
• streaming 'false' takes effect on the next replication session; no migration needed. No config or schema changes.

Checklist

• I kept the change focused and reviewable.
• I followed AGENTS.md, CLAUDE.md, and existing repository patterns.
• I added or updated tests for behavior changes, or explained why tests are not applicable.
• I updated relevant documentation, examples, migrations, or configuration.
• I did not include secrets, credentials, private customer data, unrelated refactors, or placeholder code.

🤖 Generated with Claude Code

https://claude.ai/code/session_015houJkqb8SuqPW4YpLoWCq

---

Generated by Claude Code

Summary by CodeRabbit

• New Features

  • Added dead-letter tracking metrics by projection.
  • Added decoder reset support for clean replication recovery.

• Bug Fixes

  • Reconnects now resume from the latest acknowledged position.
  • Failed records can be moved to the dead-letter queue after retries, allowing processing to continue.
  • Replication streaming transactions are explicitly rejected when unsupported.
  • Improved handling and reporting of dead-letter and sink-write failures.

• Tests

  • Added coverage for recovery positions, decoder reset behavior, streaming rejection, and dead-letter scenarios.

[cloudflare-workers-and-pages]

Deploying with    Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status	Name	Latest Commit	Updated (UTC)
❌ Deployment failed View logs	trenova	daa5c10	Aug 30 2026, 07:58 PM

[coderabbitai]

Warning

Review limit reached

Next included review available in 46 minutes.

View limit details

Limit details: You’ve used the included review currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 350f7324-a075-4ba6-a0bc-4eaceda9572d

📥 Commits

Reviewing files that changed from the base of the PR and between fcbcec1 and daa5c10.

📒 Files selected for processing (5)

• services/tms/internal/infrastructure/postgres/migrations/20260928100000_shipment_auto_rating.tx.down.sql
• services/tms/internal/infrastructure/postgres/migrations/20260928100000_shipment_auto_rating.tx.up.sql
• services/tms/internal/infrastructure/sqlite/migrations/20260928100000_shipment_auto_rating.tx.down.sql
• services/tms/internal/infrastructure/sqlite/migrations/20260928100000_shipment_auto_rating.tx.up.sql
• services/tms/pkg/reportcatalog/catalog_gen.go

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 90337bfd-ceea-4840-8dc4-057e45c9b639

📥 Commits

Reviewing files that changed from the base of the PR and between 0706b59 and fcbcec1.

📒 Files selected for processing (7)

• services/gtc/internal/adapters/primary/wal/decoder.go
• services/gtc/internal/adapters/primary/wal/decoder_test.go
• services/gtc/internal/adapters/primary/wal/reader.go
• services/gtc/internal/adapters/primary/wal/reader_test.go
• services/gtc/internal/core/services/runtime.go
• services/gtc/internal/core/services/runtime_test.go
• services/gtc/internal/infrastructure/metrics/metrics.go

Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review.

---

📝 Walkthrough

Walkthrough

The change disables streamed WAL transactions, resets decoder state on reconnect, resumes replication from the latest client LSN, and adds dead-letter handling and metrics for exhausted projection retries.

Changes

WAL recovery and streaming control

Layer / File(s)	Summary
Decoder state and streaming rejection services/gtc/internal/adapters/primary/wal/decoder.go, services/gtc/internal/adapters/primary/wal/decoder_test.go	The decoder resets relation and transaction state, parses non-streamed WAL data, and rejects streamed transaction messages. Tests cover rejection and reset behavior.
Replication reconnect and LSN selection services/gtc/internal/adapters/primary/wal/reader.go, services/gtc/internal/adapters/primary/wal/reader_test.go	Replication setup resets the decoder, disables plugin streaming, and resumes from the advanced client LSN when available. Tests cover LSN selection.

Dead-letter projection handling

Layer / File(s)	Summary
Projection failure and dead-letter flow services/gtc/internal/core/services/runtime.go, services/gtc/internal/infrastructure/metrics/metrics.go	Exhausted projection failures are written to the dead-letter queue when configured. Successful parking allows processing to continue. Sink, retry, and dead-letter metrics are updated.
Dead-letter and replay validation services/gtc/internal/core/services/runtime_test.go	Tests cover checkpoint advancement, missing or failing dead-letter queues, successful parking, and replay failures without re-parking.

Estimated code review effort: 4 (Complex) | ~45 minutes

Merge Risk: 🔵 Low · up to fcbce

The PR improves CDC recovery and dead-letter handling, but a partial dead-letter replay failure can repeat events already written to append-only sinks. The change is mergeable with owner awareness and a follow-up to ensure replay consumers or sinks provide idempotency.

Sequence Diagram(s)

sequenceDiagram
  participant Runtime
  participant Sink
  participant DeadLetterQueue
  participant Metrics
  Runtime->>Sink: Write projection
  Sink-->>Runtime: Return failure
  Runtime->>DeadLetterQueue: Write dead-letter record
  Runtime->>Metrics: Increment dead-letter counter
  DeadLetterQueue-->>Runtime: Confirm write
  Runtime->>Runtime: Continue processing

Loading

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 15 functions across 7 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main changes: preventing CDC data loss from poison records, disabling unsupported streamed transactions, and improving reconnect behavior.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches 💡 2

📝 Generate docstrings 💡

• Create stacked PR
• Commit on current branch

🛠️ Fix failing CI checks 💡

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch claude/codebase-incomplete-review-b1j99m

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[emoss08]

The two failing checks on the first push (Codegen Checks, Integration Tests) are pre-existing base-branch failures, not caused by this PR's GTC-only diff — Test Go has been red on master since Aug 24 (runs 592–603, including on base commit 0706b59eb). Root cause and the fix ported into this PR as daa5c10bf:

Migration name collision (Integration Tests). 20260928000000_shipment_auto_rating and 20260928000000_widen_rate_agreement_versions share the same numeric prefix. Bun's migrator keys migrations by that prefix only (getOrCreateMigration in bun/migrate), so during Discover the widen migration's Up/Down silently overwrote the auto-rating one — auto_rated/auto_rated_at were never created, and every seed-dependent integration test failed with column "auto_rated" of relation "shipments" does not exist (Postgres and SQLite alike). Fix: renamed shipment_auto_rating to 20260928100000 in both dialects. Databases that already recorded 20260928000000 actually executed the widen SQL, so their history stays truthful; the renamed migration applies as new, and its Postgres DDL is IF NOT EXISTS-idempotent. Reproduced TestFullSeedRunOnSQLite red locally before the rename and green after.

Stale report catalog (Codegen Checks). pkg/reportcatalog/catalog_gen.go was missing the autoRated/autoRatedAt fields from the same feature; regenerated with go generate ./internal/infrastructure/database/reportcatalog/... — the diff matches exactly what the CI check printed.

The Cloudflare Workers deployment failure looks unrelated to this diff (no client changes here) and its logs are only visible in the Cloudflare dashboard — worth a look there if it persists on this PR's latest commit.

---

Generated by Claude Code