enhance(meta): stop storing every replica endpoint twice in segment metadata - #282
Open
tinswzy wants to merge 1 commit into
Open
enhance(meta): stop storing every replica endpoint twice in segment metadata#282tinswzy wants to merge 1 commit into
tinswzy wants to merge 1 commit into
Conversation
…etadata Every segment inlines its quorum, and inside that quorum each replica's address was stored twice: once in QuorumInfo.nodes[], and again in QuorumInfo.replicas[].endpoint. The second copy existed only so the two lists could be joined by address — quorumPlacements built a map[endpoint]*QuorumNode and every caller then looked up placements[nodes[i]]. But the lists are already paired: every selection path in quorum/discovery.go appends to nodes and replicas in lockstep, and the cross-region trim shuffles and truncates both together. The address-keyed join was buying nothing, and the copy that paid for it is 282 B of every 874 B segment metadata value under Kubernetes FQDN naming — a third of the whole record, on every segment, for as long as it is retained. Mark QuorumNode.endpoint deprecated, stop writing it, and pair by index. Reading stays compatible in both directions. quorumPlacements now returns a slice aligned with nodes[] and falls back to the address join for the two legacy shapes that need it: a replica list shorter than the node list (placement was not recorded for every node — the shape the existing scope tests cover), and, defensively, one whose stored endpoints contradict the positional pairing. An older client reading a segment written by a newer one simply finds no endpoints, so every replica reads as "placement unknown": AZ-affinity read ordering falls back to the existing order and the cross-AZ metric loses a label. Correctness is untouched, because nodes[] — which the whole runtime uses to reach replicas — is unchanged. Deprecating nodes[] instead would have saved the same bytes at much higher risk: it has 57 references across 7 files on the append, cleanup, compacted-mark, fence and delete paths; its index is the replica identity behind channelErrors[i], the ack/failure bitsets and recordReplicaResult; and four fallback sites key off len(quorum.Nodes) == 0 to substitute 127.0.0.1, so a segment written without nodes would have its quorum silently replaced by the loopback address — the failure already seen in #216. Fixes #280 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #282 +/- ##
==========================================
- Coverage 83.80% 83.76% -0.05%
==========================================
Files 183 183
Lines 25819 25832 +13
==========================================
Hits 21638 21638
- Misses 3193 3204 +11
- Partials 988 990 +2
🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #280.
What
Every segment inlines its quorum, and inside that quorum each replica's address was stored twice: once in
QuorumInfo.nodes[], and again inQuorumInfo.replicas[].endpoint.The second copy existed only as a join key —
quorumPlacementsbuilt amap[endpoint]*QuorumNodeand every caller then looked upplacements[nodes[i]]. But the two lists are already paired: every selection path inquorum/discovery.goappends tonodesandreplicasin lockstep, and the cross-region trim shuffles and truncates both together. The address-keyed join was buying nothing.This marks
QuorumNode.endpointdeprecated, stops writing it, and pairs the lists by index.Measured
One
SegmentMetadata, Es=3, endpoint = K8s StatefulSet pod FQDN (wp-…-0.wp-…-headless.<ns>.svc.cluster.local:18080, 92 B):For scale: at the retention floor implied by
segmentRollingPolicy.maxInterval: 600s, an idle log still produces 6 segments/hour, so a 72h window is 432 segments per log and a 7-day window is 1008 — every one of them carrying that copy for as long as it is retained.Compatibility
Both directions degrade safely, because
nodes[]— which the entire runtime uses to reach replicas — is unchanged.New client, old segment.
quorumPlacementsnow returns a slice aligned withnodes[], and falls back to the address join for the two legacy shapes that need it:scopeTestQuorumin the existing tests already covers);Old client, new segment. It finds no endpoints, so every replica reads as "placement unknown": AZ-affinity read ordering falls back to the existing order, and the cross-AZ metric loses a label. No correctness impact.
Why not deprecate
nodes[]insteadSame bytes saved, much higher risk:
QuorumInfo.nodeshas 57 non-test references across 7 files — append, batch append, cleanup, compacted-mark distribution, fence, delete.QuorumInfo.replicashas 2.nodes[i]is the replica identity:channelErrors[i], the ack/failure bitsets,recordReplicaResult(serverIndex)andHandleAppendRequestFailure(..., serverIndex, ...)are all indexed by position innodes.127.0.0.1landmine. Four fallback sites key offlen(quorum.Nodes) == 0and substitute the loopback address. A segment written withoutnodeswould have its healthy 3-replica quorum silently replaced — the failure already observed in bug: service-mode recovery falls back to 127.0.0.1 for old WAL segment metadata without QuorumInfo #216.Tests
New:
woodpecker/segment/quorum_placement_test.go— index alignment when the endpoint is omitted, across all three consumers (quorumNodeScopes,activeSegmentNodes,orderedQuorumReadCandidates), plus a guard that a legacy endpoint contradicting the positional pairing still wins.woodpecker/quorum/quorum_node_test.go—quorumNodeFromMetano longer writes the endpoint while keeping the rest of the topology, and cross-region selection hands back replicas index-aligned with nodes.Changed:
TestQuorumDiscovery_SelectQuorumNodes_SingleRegion_SuccessassertedReplicas[0].Endpoint, which is the behavior being removed; it now asserts the index pairing instead.The existing scope tests (
quorum_scope_test.go) are unchanged and still pass — they exercise the legacy address-join path.Follow-ups (not in this PR)
SegmentCompactedNotifyStatus.quorum_notify_statusis keyed by the full endpoint, making themarking/record 314 B instead of 92 B under FQDN naming;node_idwould be smaller and a better fit for "has this node acked the mark".quorums/<quorumId>andStoreQuorumInfo/GetQuorumInfoalready exist but no production path calls them.Verification
go test -race -shortover every non-tests/package, run in chunks locally:./woodpecker/...,./server/...,./meta/...,./proto/...,./cmd/...,./common/...,./mocks/...— all pass.common/objectstorageTestNewAzureObjectStorageClient_IAM_InvalidCredentialshangs on a live azblobGetPropertiescall and times out. It fails identically on an unmodifiedmasterworktree, so it is a pre-existing environment dependency, not a regression from this change.make protowas verified byte-stable on a clean tree before the proto edit, so themeta.pb.godiff is entirely the deprecation marker, the new comments, and the raw-descriptor reflow — no unrelated codegen drift.QuorumNodeis the only struct touched.