Skip to content

Add a UTxO RPC (gRPC) chain producer - #5

Draft
Jimbo4350 wants to merge 2 commits into
masterfrom
feat/cardano-rpc-chain-sync
Draft

Add a UTxO RPC (gRPC) chain producer#5
Jimbo4350 wants to merge 2 commits into
masterfrom
feat/cardano-rpc-chain-sync

Conversation

@Jimbo4350

Copy link
Copy Markdown
Contributor

Draft: builds and unit-tests pass, but has never been run against a live cardano-rpc server. See "What is not done" below.

Context

Sieve has had one block source: a local node over node-to-client ChainSync. cardano-rpc now serves the UTxO RPC SyncService, and its FollowTip method — despite the name — replays history from an intersection point, so it can drive a full sync rather than only tail the tip.

This adds it as a second producer, selected by --rpc-socket-path, --rpc-address, or --via-rpc (which derives rpc.sock beside --socket-path, cardano-rpc's own default).

The original design note in Cardano/Sieve.hs always anticipated this: "chain-sync or ogmios or cardano-rpc to get blocks — This needs a wrapper around it to switch between the different sources of blocks."

Shape

New modules under Cardano.Sieve.CardanoRpc, a sibling of Cardano.Sieve.Node rather than a child, because only the transport differs:

module job
Decode native_bytesBlockInMode
Endpoint where the server is; grapesy connection setup
Response protobuf projections, so the loop reads as chain logic
Follow the stream loop

Both producers share sieveBlock, startPoints and all of Node.Insert, so they write identical databases.

The one genuinely new capability

Over ChainSync, cardano-api decodes inside connectToLocalNode and sieve is handed a BlockInMode. Over gRPC the client gets AnyChainBlock.native_bytes and must decode them itself — sieve had no ByteString -> BlockInMode anywhere.

Those bytes are the ChainDB's on-disk serialisation (GetRawBlock), so the consensus disk codec applies, and the era tag it carries makes one decoder work across every era. Taking the raw bytes rather than the parsed protobuf block in the same message is what keeps Node.Decode/Node.Encode reusable, and with them the ledger-exact semantics they encode.

Two things the gRPC path gets for free

A real at-the-tip signal. Every response carries the node's live tip, so "am I at the tip?" is asked and answered rather than inferred from an empty pipeline as collectFlushingWhenIdle must. One comparison drives both the deferred index build and the flush cadence.

A continuous decoder cross-check. Every block arrives with the slot and hash the server derived from the same bytes. The loop compares them on every block, so the decoder proves itself rather than being trusted — which matters while the post-Byron arms have no golden fixtures.

How to trust this PR

cabal build -j4 clean with no warnings; fourmolu applied; all 57 tests pass.

The decoder is tested against real mainnet Byron blocks, borrowed from cardano-rpc's own fixtures. Byron is the era worth testing because EpochSlots is consumed on its decoder arms and nowhere else — the epoch-boundary block landing on slot 2,160,000 is only correct if slots-per-epoch really is 21,600. Trailing bytes, truncation, non-CBOR and an unknown era tag are all asserted to fail.

CLI validation verified by hand:

$ cardano-sieve --rpc-socket-path /tmp/rpc.sock --database x --until 100
cardano-sieve: --until is not supported when syncing over UTxO RPC

$ cardano-sieve --via-rpc --database x
cardano-sieve: --via-rpc needs --socket-path to derive the rpc socket path

What is not done

  • Never run against a live server. No block has actually travelled this path.
  • The differential test is the real acceptance test and has not been run: sync the same range over both producers and diff outputs/unspent/blocks. Anything but identical counts means the port is wrong.
  • Post-Byron decoder arms have no fixtures — covered only by the runtime cross-check. Capturing one Conway block's native_bytes would close this.
  • Rollback is untested. Bounded historical ranges cannot exercise it, same gap the ChainSync path has.
  • --until is rejected over gRPC (FollowTip has no stopping condition), so the bench scripts cannot yet compare the two producers.

Incidental

test/Main.hs did not compile against the pinned cardano-api — AdaAssetId is now a constructor of AssetId rather than a standalone export. Fixed here because nothing could be tested otherwise; happy to split it out.

sieveBlock, startPoints and describeSelectors become exported from Node.Fetch rather than duplicated — each encodes a policy that must not differ between producers. No logic in Fetch.hs moved.

🤖 Generated with Claude Code

Sieve has had one block source: a local node over node-to-client
ChainSync. cardano-rpc now serves the UTxO RPC SyncService, whose
FollowTip method replays history from an intersection point rather than
only following the tip, so it can drive a full sync. This adds it as a
second producer, selected by --rpc-socket-path, --rpc-address or
--via-rpc.

The new modules live under Cardano.Sieve.CardanoRpc, alongside
Cardano.Sieve.Node rather than inside it, because only the transport
differs: both producers share sieveBlock, startPoints and the whole of
Node.Insert, so they write identical databases.

Decode.hs is the only genuinely new capability. Over ChainSync,
cardano-api decodes blocks inside connectToLocalNode and sieve is handed
a BlockInMode; over gRPC the client receives AnyChainBlock.native_bytes
and must decode them itself. Those bytes are the ChainDB's on-disk
serialisation (GetRawBlock), so the consensus disk codec applies and the
era tag it carries makes one decoder work for every era. Taking the raw
bytes rather than the parsed protobuf block in the same message is what
keeps Node.Decode and Node.Encode reusable, and with them the
ledger-exact semantics they encode.

Two things the gRPC path gets for free. Every response carries the node's
live tip, so "am I at the tip?" is asked and answered rather than
inferred from an empty pipeline as collectFlushingWhenIdle must — that
one comparison drives both the deferred index build and the flush
cadence. And every block arrives with the slot and hash the server
derived from the same bytes, so the decoder is cross-checked on every
block, which matters while the post-Byron arms have no golden fixtures.

sieveBlock, startPoints and describeSelectors become exported from
Node.Fetch rather than duplicated: each encodes a policy that must not
differ between producers. No logic in Fetch.hs moved.

--until is rejected over gRPC, in invocationOf so it fails before the
database is opened: FollowTip has no stopping condition of its own.

Also fixes an unrelated breakage in test/Main.hs, which did not compile
against the pinned cardano-api: AdaAssetId is now a constructor of
AssetId rather than a standalone export.

Tests cover the decoder against real mainnet Byron blocks, borrowed from
cardano-rpc's own fixtures. Byron is the era worth testing here because
EpochSlots is consumed on its decoder arms and nowhere else; the
epoch-boundary block landing on slot 2,160,000 is only correct if
slots-per-epoch really is 21,600.
LambdaCase where it was assumed; Proxy from Data.Proxy rather than
grapesy; defMessage via grapesy's Protobuf re-export instead of a direct
proto-lens dependency.

Also corrects the HTTP/2 window comment, which claimed 256KiB was small
against a mainnet block. Mainnet caps a block at 90,112 + 1,100 bytes,
so the default window holds about three of them. The 8MiB choice stands
but for a different reason: it puts roughly the same order of work in
flight as ChainSync's fifty-block pipeline.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant