Add a UTxO RPC (gRPC) chain producer - #5
Draft
Jimbo4350 wants to merge 2 commits into
Draft
Conversation
Sieve has had one block source: a local node over node-to-client ChainSync. cardano-rpc now serves the UTxO RPC SyncService, whose FollowTip method replays history from an intersection point rather than only following the tip, so it can drive a full sync. This adds it as a second producer, selected by --rpc-socket-path, --rpc-address or --via-rpc. The new modules live under Cardano.Sieve.CardanoRpc, alongside Cardano.Sieve.Node rather than inside it, because only the transport differs: both producers share sieveBlock, startPoints and the whole of Node.Insert, so they write identical databases. Decode.hs is the only genuinely new capability. Over ChainSync, cardano-api decodes blocks inside connectToLocalNode and sieve is handed a BlockInMode; over gRPC the client receives AnyChainBlock.native_bytes and must decode them itself. Those bytes are the ChainDB's on-disk serialisation (GetRawBlock), so the consensus disk codec applies and the era tag it carries makes one decoder work for every era. Taking the raw bytes rather than the parsed protobuf block in the same message is what keeps Node.Decode and Node.Encode reusable, and with them the ledger-exact semantics they encode. Two things the gRPC path gets for free. Every response carries the node's live tip, so "am I at the tip?" is asked and answered rather than inferred from an empty pipeline as collectFlushingWhenIdle must — that one comparison drives both the deferred index build and the flush cadence. And every block arrives with the slot and hash the server derived from the same bytes, so the decoder is cross-checked on every block, which matters while the post-Byron arms have no golden fixtures. sieveBlock, startPoints and describeSelectors become exported from Node.Fetch rather than duplicated: each encodes a policy that must not differ between producers. No logic in Fetch.hs moved. --until is rejected over gRPC, in invocationOf so it fails before the database is opened: FollowTip has no stopping condition of its own. Also fixes an unrelated breakage in test/Main.hs, which did not compile against the pinned cardano-api: AdaAssetId is now a constructor of AssetId rather than a standalone export. Tests cover the decoder against real mainnet Byron blocks, borrowed from cardano-rpc's own fixtures. Byron is the era worth testing here because EpochSlots is consumed on its decoder arms and nowhere else; the epoch-boundary block landing on slot 2,160,000 is only correct if slots-per-epoch really is 21,600.
LambdaCase where it was assumed; Proxy from Data.Proxy rather than grapesy; defMessage via grapesy's Protobuf re-export instead of a direct proto-lens dependency. Also corrects the HTTP/2 window comment, which claimed 256KiB was small against a mainnet block. Mainnet caps a block at 90,112 + 1,100 bytes, so the default window holds about three of them. The 8MiB choice stands but for a different reason: it puts roughly the same order of work in flight as ChainSync's fifty-block pipeline.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft: builds and unit-tests pass, but has never been run against a live cardano-rpc server. See "What is not done" below.
Context
Sieve has had one block source: a local node over node-to-client ChainSync.
cardano-rpcnow serves the UTxO RPCSyncService, and itsFollowTipmethod — despite the name — replays history from an intersection point, so it can drive a full sync rather than only tail the tip.This adds it as a second producer, selected by
--rpc-socket-path,--rpc-address, or--via-rpc(which derivesrpc.sockbeside--socket-path, cardano-rpc's own default).The original design note in
Cardano/Sieve.hsalways anticipated this: "chain-sync or ogmios or cardano-rpc to get blocks — This needs a wrapper around it to switch between the different sources of blocks."Shape
New modules under
Cardano.Sieve.CardanoRpc, a sibling ofCardano.Sieve.Noderather than a child, because only the transport differs:Decodenative_bytes→BlockInModeEndpointResponseFollowBoth producers share
sieveBlock,startPointsand all ofNode.Insert, so they write identical databases.The one genuinely new capability
Over ChainSync, cardano-api decodes inside
connectToLocalNodeand sieve is handed aBlockInMode. Over gRPC the client getsAnyChainBlock.native_bytesand must decode them itself — sieve had noByteString -> BlockInModeanywhere.Those bytes are the ChainDB's on-disk serialisation (
GetRawBlock), so the consensus disk codec applies, and the era tag it carries makes one decoder work across every era. Taking the raw bytes rather than the parsed protobuf block in the same message is what keepsNode.Decode/Node.Encodereusable, and with them the ledger-exact semantics they encode.Two things the gRPC path gets for free
A real at-the-tip signal. Every response carries the node's live tip, so "am I at the tip?" is asked and answered rather than inferred from an empty pipeline as
collectFlushingWhenIdlemust. One comparison drives both the deferred index build and the flush cadence.A continuous decoder cross-check. Every block arrives with the slot and hash the server derived from the same bytes. The loop compares them on every block, so the decoder proves itself rather than being trusted — which matters while the post-Byron arms have no golden fixtures.
How to trust this PR
cabal build -j4clean with no warnings;fourmoluapplied; all 57 tests pass.The decoder is tested against real mainnet Byron blocks, borrowed from cardano-rpc's own fixtures. Byron is the era worth testing because
EpochSlotsis consumed on its decoder arms and nowhere else — the epoch-boundary block landing on slot 2,160,000 is only correct if slots-per-epoch really is 21,600. Trailing bytes, truncation, non-CBOR and an unknown era tag are all asserted to fail.CLI validation verified by hand:
What is not done
outputs/unspent/blocks. Anything but identical counts means the port is wrong.native_byteswould close this.--untilis rejected over gRPC (FollowTiphas no stopping condition), so the bench scripts cannot yet compare the two producers.Incidental
test/Main.hsdid not compile against the pinned cardano-api —AdaAssetIdis now a constructor ofAssetIdrather than a standalone export. Fixed here because nothing could be tested otherwise; happy to split it out.sieveBlock,startPointsanddescribeSelectorsbecome exported fromNode.Fetchrather than duplicated — each encodes a policy that must not differ between producers. No logic inFetch.hsmoved.🤖 Generated with Claude Code