fix(gtc): clear projected data when a source table is truncated by emoss08 · Pull Request #564 · emoss08/Trenova

emoss08 · 2026-08-31T19:26:46Z

Description

The WAL decoder already emitted TRUNCATE records, but no sink handled them. A TRUNCATE on a projected table had two failure modes: the Meilisearch sink could not derive a document key from the data-less record, so every truncate burned its retries and landed in the DLQ; the Redis JSON sink fell through to JSON.SET with a key rendered from empty placeholders. Either way, all previously projected documents and cache keys stayed behind as stale data.

Per-sink handling now:

Meilisearch — deletes the projection's documents via DeleteDocumentsByFilter on the _projection/_source_table metadata every document already carries. Those two fields are now always registered as filterable (previously filterable attributes were only configured when the projection declared some), which also makes the delete correct if two projections ever share an index.
Redis JSON — renders the key template into a glob pattern (field placeholders become *, literal text is glob-escaped, .Schema/.Table render normally) and deletes matching keys via SCAN + batched UNLINK. A pattern with no literal content is refused outright, so a misconfigured all-placeholder template can never scan-and-delete the entire keyspace.
Redis stream / TCA stream — unchanged on purpose: stream names are static and forwarding the TRUNCATE event to consumers is the correct semantic there.

Related Issue or Discussion

Follow-up from the codebase audit driving PRs #557–#563 (gtc reliability slice).

Type of Change

Scope

services/gtc/internal/adapters/secondary/redis/template.go — wildcard-pattern rendering with glob escaping and the no-literal-anchor guard
services/gtc/internal/adapters/secondary/redis/sink.go — truncate branch in the JSON sink (SCAN + batched UNLINK), template cache refactor
services/gtc/internal/adapters/secondary/meilisearch/sink.go — truncate branch (delete-by-filter), always-filterable metadata fields
New/extended tests in both packages; go.mod/go.sum add miniredis (test-only)

Validation

cd services/gtc && go build ./... && go vet ./... && go test ./... — all green
Truncate sink tests demonstrated red against the pre-change code (old fall-through path fails with "projection payload is empty"), then green with the fix
gofmt -l clean; new lines hand-checked against the 100-column golines budget
cd services/tms && task test / task lint — not run; no TMS changes
cd client && pnpm build / pnpm lint — not run; no client changes

Deployment Notes

On the first write per Meilisearch index after deploy, filterable attributes are updated to include _projection/_source_table, which triggers a one-time reindex task per index.
Documents written before this change already carry the metadata fields, so truncates work on pre-existing documents once the attributes become filterable.
New test-only dependency: github.com/alicebob/miniredis/v2 (the repo had no Redis fake; the truncate path is a destructive glob delete and is now covered end to end).

Checklist

I kept the change focused and reviewable.
I followed AGENTS.md, CLAUDE.md, and existing repository patterns.
I added or updated tests for behavior changes, or explained why tests are not applicable.
I updated relevant documentation, examples, migrations, or configuration.
I did not include secrets, credentials, private customer data, unrelated refactors, or placeholder code.

🤖 Generated with Claude Code

https://claude.ai/code/session_015houJkqb8SuqPW4YpLoWCq

Generated by Claude Code

Summary by CodeRabbit

New Features
- Added support for truncating projection data in Meilisearch and Redis.
- Redis truncation removes matching keys in batches while preserving unrelated data.
- Meilisearch filtering now consistently includes projection and source metadata.
Bug Fixes
- Improved handling of wildcard key patterns, including escaping special characters and rejecting unsafe patterns.
Tests
- Added coverage for truncation, filtering, wildcard patterns, batching, and preservation of unrelated records.

The WAL decoder already emitted TRUNCATE records, but no sink handled them: Meilisearch could not derive a document key from the data-less record (every truncate dead-lettered after retries) and the Redis JSON sink fell through to JSON.SET with a broken key, so truncated tables left all projected documents and cache keys stale. Meilisearch now deletes the projection's documents by filter on the _projection/_source_table metadata every document already carries, with those fields always registered as filterable so the delete works on shared indexes too. The Redis JSON sink renders the key template into a glob-escaped MATCH pattern (placeholders become wildcards) and removes matching keys via SCAN+UNLINK in batches, refusing patterns with no literal anchor so a truncate can never wipe the whole keyspace. Stream sinks already forward TRUNCATE events to consumers and are unchanged. Adds miniredis as a test dependency to cover the destructive delete path end to end. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015houJkqb8SuqPW4YpLoWCq

coderabbitai · 2026-08-31T19:27:39Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 86be04f0-c8db-4518-9723-6f6783e7e9c1

📥 Commits

Reviewing files that changed from the base of the PR and between 4851f67 and 4add5da.

⛔ Files ignored due to path filters (1)

services/gtc/go.sum is excluded by !**/*.sum

📒 Files selected for processing (7)

services/gtc/go.mod
services/gtc/internal/adapters/secondary/meilisearch/sink.go
services/gtc/internal/adapters/secondary/meilisearch/sink_test.go
services/gtc/internal/adapters/secondary/redis/sink.go
services/gtc/internal/adapters/secondary/redis/sink_test.go
services/gtc/internal/adapters/secondary/redis/template.go
services/gtc/internal/adapters/secondary/redis/template_test.go

Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review.

📝 Walkthrough

Walkthrough

The PR adds projection-scoped truncate handling to Meilisearch and Redis sinks. Redis truncation generates escaped wildcard patterns, scans matching keys, and deletes them in batches. Meilisearch truncation filters documents by projection metadata.

Changes

Projection truncation

Layer / File(s)	Summary
Redis wildcard template patterns `services/gtc/internal/adapters/secondary/redis/template.go`, `services/gtc/internal/adapters/secondary/redis/template_test.go`, `services/gtc/go.mod`	Redis templates now generate escaped wildcard patterns. Templates without literal anchors return an error.
Redis projection truncation `services/gtc/internal/adapters/secondary/redis/sink.go`, `services/gtc/internal/adapters/secondary/redis/sink_test.go`	Truncate operations scan projection-matching keys and remove them with batched `UNLINK` calls. Tests cover filtering, batching, unrelated keys, and invalid templates.
Meilisearch projection truncation `services/gtc/internal/adapters/secondary/meilisearch/sink.go`, `services/gtc/internal/adapters/secondary/meilisearch/sink_test.go`	Truncate operations delete documents by `_projection` and `_source_table`. These metadata fields are always configured as filterable attributes.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Merge Risk: 🟠 High · up to 4add5

This PR changes Redis TRUNCATE handling to delete keys by wildcard pattern. Overlapping projection templates could remove another projection's keys, while interrupted or concurrent cleanup could leave inconsistent data, so the change is not merge-ready without stronger namespace isolation and cleanup convergence guarantees.

Sequence Diagram(s)

sequenceDiagram
  participant SourceRecord
  participant RedisSink
  participant Template
  participant Redis
  SourceRecord->>RedisSink: OperationTruncate
  RedisSink->>Template: WildcardPattern(record, primaryKeys)
  Template-->>RedisSink: Projection-scoped key pattern
  RedisSink->>Redis: Scan matching keys
  RedisSink->>Redis: UNLINK keys in batches

sequenceDiagram
  participant SourceRecord
  participant MeilisearchSink
  participant Meilisearch
  SourceRecord->>MeilisearchSink: OperationTruncate
  MeilisearchSink->>Meilisearch: Delete documents by projection and source-table filter
  Meilisearch-->>MeilisearchSink: Task identifier
  MeilisearchSink->>Meilisearch: Wait for truncate task

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 26 functions across 6 files. (1 skipped: 1…	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely describes the main change: clearing projected data when a source table is truncated.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

Full details: Docstring Coverage

Explanation

Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 26 functions across 6 files. (1 skipped: 1 unsupported.)

Fix all pre-merge checks with AI

✨ Finishing Touches 💡 2

📝 Generate docstrings 💡

Create stacked PR
Commit on current branch

🛠️ Fix failing CI checks 💡

Create stacked PR
Commit on current branch

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch claude/gtc-truncate-b1j99m

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

socket-security · 2026-08-31T19:27:45Z

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Vulnerability	Quality	Maintenance	License
	golang/github.com/alicebob/miniredis/v2@v2.38.0

View full report

cloudflare-workers-and-pages · 2026-08-31T19:36:14Z

Deploying with Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status	Name	Latest Commit	Updated (UTC)
❌ Deployment failed View logs	trenova	`4add5da`	Aug 31 2026, 06:33 PM

emoss08 · 2026-08-31T19:36:31Z

Standing down on the Workers Builds: trenova failure: this PR touches only Go code in services/gtc (plus its go.mod/go.sum), which the Cloudflare Workers client deploy does not build or consume. The same check has failed on every recent commit regardless of content — including Go-only PRs #557–#563 and their merges to master — so it's a pre-existing environment/configuration issue in the Cloudflare build, not something this diff introduces or can fix. Build logs are only visible in the Cloudflare dashboard, so no fix can be ported from here.

All repository CI on this head (Lint, Build, Unit Tests, Integration Tests, Codegen Checks) is green.

Generated by Claude Code

Navigation Menu

fix(gtc): clear projected data when a source table is truncated - #564

fix(gtc): clear projected data when a source table is truncated#564
emoss08 wants to merge 1 commit into
masterfrom
claude/gtc-truncate-b1j99m

emoss08 commented Aug 31, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

coderabbitai Bot commented Aug 31, 2026 •

edited

Loading

Walkthrough

Changes

Sequence Diagram(s)

❌ Failed checks (1 warning)

Uh oh!

socket-security Bot commented Aug 31, 2026

Uh oh!

cloudflare-workers-and-pages Bot commented Aug 31, 2026

Uh oh!

emoss08 commented Aug 31, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

emoss08 commented Aug 31, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issue or Discussion

Type of Change

Scope

Validation

Deployment Notes

Checklist

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented Aug 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Sequence Diagram(s)

❌ Failed checks (1 warning)

Uh oh!

socket-security Bot commented Aug 31, 2026

Uh oh!

cloudflare-workers-and-pages Bot commented Aug 31, 2026

Deploying with Cloudflare Workers

Uh oh!

emoss08 commented Aug 31, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

emoss08 commented Aug 31, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Aug 31, 2026 •

edited

Loading