[JAM parachain-service] Implement additional data support

[alindima]

Implements paritytech/parachain-service#14

Completes the implementation started on the parchain-service branch.

V3 parachains now rely on this new data transport for the relay state proof

[bkchr]

I think this goes into the right direction, but I would do some conceptual changes.

1. sp-additional-data should only expose generic/shared host functions, types and constants. E.g. only the finalize host functions and nothing around read_relay_state.
2. cumulus-client-addtional-data should be changed to host all the relevant bits and pieces for reading the relay state.

trait RelayState {
     fn read(key: Vec<u8>) -> Option<Vec<u8>> {
              self.get_extension<RelayStateExt>()?.read(key)
     }
}

sp-addiotional-data should expose some AdditionalDataExt (this is for example used by finalize to generate the hash). This extension should just contain "pointers" to all the other extensions that also host additonal data. So, that it can later on all self.all_additional_datas.iter().map(|dp| dp.finalize()).fold_into_hash().

[bkchr]

Which are unused?

[alindima]

I don't remember, there was some CI test failing because of this. But there were many of them unused. Would be a good idea to get the CI in a running state on the current state of the parchain-service branch before merging more stuff on top

[bkchr]

I would expect that this is specific for only reading relay chain state.

[alindima]

for now, but maybe follow-up use cases will emerge

[alindima]

sp-additional-data should only expose generic/shared host functions, types and constants. E.g. only the finalize host functions and nothing around read_relay_state.

I agree that the substrate crate should be kept generic. The reality is that the finalize host function also would need to have some logic specific to each particular item in the additional data. That's up to each such "extension" to determine.
For example, for they relay/JAM state proof, we also check that the state proof does not contain keys that are not needed during block execution. But this is a particular thing for parachains, not a general substrate thing and not all such extensions will need to do this.
This is what I was talking about here: paritytech/parachain-service#14 (comment)

So IMO, it makes sense to either:

1. Define both the read and finalise host functions in substrate, and have them called with a string identifier (having overall two host function calls that drive the entire additional data onchain interaction). This would call into an extension defined in substrate that wraps over a generic dyn AdditionalDataProvider that the parachain implements and registers.
2. Remove the host functions from the substrate crate. So the only thing remaining there will be the pub type AdditionalData = BTreeMap<String, Vec<u8>>; type definition. Move the entire host function definition in cumulus

sp-addiotional-data should expose some AdditionalDataExt (this is for example used by finalize to generate the hash). This extension should just contain "pointers" to all the other extensions that also host additonal data. So, that it can later on all self.all_additional_datas.iter().map(|dp| dp.finalize()).fold_into_hash().

This sounds very similar to the idea number 1 above. Do you agree that it'd then make sense to also have a common read host function that dispatches based on some param identifier?

[bkchr]

1. Define both the read and finalise host functions in substrate, and have them called with a string identifier (having overall two host function calls that drive the entire additional data onchain interaction). This would call into an extension defined in substrate that wraps over a generic dyn AdditionalDataProvider that the parachain implements and registers.

No, we should not add any strings there. We just need some generic AdditionalDataProvider or whatever trait that provides the finalize method. finalize would call this method for all additional data providers. And in the case of e.g. the relay state reading, it returns an Err if not all values got read. (The error is not forwarded to the runtime, we could just panic on the node side)

[paritytech-workflow-stopper]

All GitHub workflows were cancelled due to failure one of the required jobs.
Failed workflow url: https://github.com/paritytech/polkadot-sdk/actions/runs/33635139602
Failed job name: quick-benchmarks-omni