Skip to content

enhance(meta): stop storing every replica endpoint twice in segment metadata - #282

Open
tinswzy wants to merge 1 commit into
masterfrom
enhance/dedup-quorum-endpoint
Open

enhance(meta): stop storing every replica endpoint twice in segment metadata#282
tinswzy wants to merge 1 commit into
masterfrom
enhance/dedup-quorum-endpoint

Conversation

@tinswzy

@tinswzy tinswzy commented Aug 28, 2026

Copy link
Copy Markdown
Collaborator

Fixes #280.

What

Every segment inlines its quorum, and inside that quorum each replica's address was stored twice: once in QuorumInfo.nodes[], and again in QuorumInfo.replicas[].endpoint.

The second copy existed only as a join key — quorumPlacements built a map[endpoint]*QuorumNode and every caller then looked up placements[nodes[i]]. But the two lists are already paired: every selection path in quorum/discovery.go appends to nodes and replicas in lockstep, and the cross-region trim shuffles and truncates both together. The address-keyed join was buying nothing.

This marks QuorumNode.endpoint deprecated, stops writing it, and pairs the lists by index.

Measured

One SegmentMetadata, Es=3, endpoint = K8s StatefulSet pod FQDN (wp-…-0.wp-…-headless.<ns>.svc.cluster.local:18080, 92 B):

bytes
before (endpoint written) 874
after (endpoint omitted) 589
saved 285 (33%)

For scale: at the retention floor implied by segmentRollingPolicy.maxInterval: 600s, an idle log still produces 6 segments/hour, so a 72h window is 432 segments per log and a 7-day window is 1008 — every one of them carrying that copy for as long as it is retained.

Compatibility

Both directions degrade safely, because nodes[] — which the entire runtime uses to reach replicas — is unchanged.

New client, old segment. quorumPlacements now returns a slice aligned with nodes[], and falls back to the address join for the two legacy shapes that need it:

  • a replica list shorter than the node list (placement was not recorded for every node — the shape scopeTestQuorum in the existing tests already covers);
  • defensively, one whose stored endpoints contradict the positional pairing.

Old client, new segment. It finds no endpoints, so every replica reads as "placement unknown": AZ-affinity read ordering falls back to the existing order, and the cross-AZ metric loses a label. No correctness impact.

Why not deprecate nodes[] instead

Same bytes saved, much higher risk:

  • Blast radius. QuorumInfo.nodes has 57 non-test references across 7 files — append, batch append, cleanup, compacted-mark distribution, fence, delete. QuorumInfo.replicas has 2.
  • Index semantics. nodes[i] is the replica identity: channelErrors[i], the ack/failure bitsets, recordReplicaResult(serverIndex) and HandleAppendRequestFailure(..., serverIndex, ...) are all indexed by position in nodes.
  • The 127.0.0.1 landmine. Four fallback sites key off len(quorum.Nodes) == 0 and substitute the loopback address. A segment written without nodes would have its healthy 3-replica quorum silently replaced — the failure already observed in bug: service-mode recovery falls back to 127.0.0.1 for old WAL segment metadata without QuorumInfo #216.

Tests

New:

  • woodpecker/segment/quorum_placement_test.go — index alignment when the endpoint is omitted, across all three consumers (quorumNodeScopes, activeSegmentNodes, orderedQuorumReadCandidates), plus a guard that a legacy endpoint contradicting the positional pairing still wins.
  • woodpecker/quorum/quorum_node_test.goquorumNodeFromMeta no longer writes the endpoint while keeping the rest of the topology, and cross-region selection hands back replicas index-aligned with nodes.

Changed: TestQuorumDiscovery_SelectQuorumNodes_SingleRegion_Success asserted Replicas[0].Endpoint, which is the behavior being removed; it now asserts the index pairing instead.

The existing scope tests (quorum_scope_test.go) are unchanged and still pass — they exercise the legacy address-join path.

Follow-ups (not in this PR)

  • SegmentCompactedNotifyStatus.quorum_notify_status is keyed by the full endpoint, making the marking/ record 314 B instead of 92 B under FQDN naming; node_id would be smaller and a better fit for "has this node acked the mark".
  • The remaining ~838 B of every segment's metadata is quorum topology identical across every segment sharing a quorum. quorums/<quorumId> and StoreQuorumInfo/GetQuorumInfo already exist but no production path calls them.

Verification

go test -race -short over every non-tests/ package, run in chunks locally:

  • ./woodpecker/..., ./server/..., ./meta/..., ./proto/..., ./cmd/..., ./common/..., ./mocks/... — all pass.
  • One exception: common/objectstorage TestNewAzureObjectStorageClient_IAM_InvalidCredentials hangs on a live azblob GetProperties call and times out. It fails identically on an unmodified master worktree, so it is a pre-existing environment dependency, not a regression from this change.

make proto was verified byte-stable on a clean tree before the proto edit, so the meta.pb.go diff is entirely the deprecation marker, the new comments, and the raw-descriptor reflow — no unrelated codegen drift. QuorumNode is the only struct touched.

…etadata

Every segment inlines its quorum, and inside that quorum each replica's
address was stored twice: once in QuorumInfo.nodes[], and again in
QuorumInfo.replicas[].endpoint. The second copy existed only so the two
lists could be joined by address — quorumPlacements built a
map[endpoint]*QuorumNode and every caller then looked up placements[nodes[i]].

But the lists are already paired: every selection path in quorum/discovery.go
appends to nodes and replicas in lockstep, and the cross-region trim shuffles
and truncates both together. The address-keyed join was buying nothing, and
the copy that paid for it is 282 B of every 874 B segment metadata value
under Kubernetes FQDN naming — a third of the whole record, on every segment,
for as long as it is retained.

Mark QuorumNode.endpoint deprecated, stop writing it, and pair by index.

Reading stays compatible in both directions. quorumPlacements now returns a
slice aligned with nodes[] and falls back to the address join for the two
legacy shapes that need it: a replica list shorter than the node list
(placement was not recorded for every node — the shape the existing scope
tests cover), and, defensively, one whose stored endpoints contradict the
positional pairing. An older client reading a segment written by a newer one
simply finds no endpoints, so every replica reads as "placement unknown":
AZ-affinity read ordering falls back to the existing order and the cross-AZ
metric loses a label. Correctness is untouched, because nodes[] — which the
whole runtime uses to reach replicas — is unchanged.

Deprecating nodes[] instead would have saved the same bytes at much higher
risk: it has 57 references across 7 files on the append, cleanup,
compacted-mark, fence and delete paths; its index is the replica identity
behind channelErrors[i], the ack/failure bitsets and recordReplicaResult; and
four fallback sites key off len(quorum.Nodes) == 0 to substitute 127.0.0.1,
so a segment written without nodes would have its quorum silently replaced by
the loopback address — the failure already seen in #216.

Fixes #280

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@codecov

codecov Bot commented Aug 28, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 97.14286% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 83.76%. Comparing base (ccb3d57) to head (f6910a2).

Files with missing lines Patch % Lines
woodpecker/quorum/discovery.go 90.90% 0 Missing and 1 partial ⚠️
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #282      +/-   ##
==========================================
- Coverage   83.80%   83.76%   -0.05%     
==========================================
  Files         183      183              
  Lines       25819    25832      +13     
==========================================
  Hits        21638    21638              
- Misses       3193     3204      +11     
- Partials      988      990       +2     
Components Coverage Δ
Server 83.40% <ø> (-0.02%) ⬇️
Client 90.67% <97.14%> (-0.02%) ⬇️
Meta 85.69% <ø> (-0.50%) ⬇️
Common 88.73% <ø> (-0.04%) ⬇️
Files with missing lines Coverage Δ
woodpecker/segment/segment_handle.go 92.06% <100.00%> (+0.09%) ⬆️
woodpecker/quorum/discovery.go 97.92% <90.90%> (+0.87%) ⬆️

... and 8 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

enhance(meta): QuorumInfo stores every replica endpoint twice — deprecate QuorumNode.endpoint

1 participant