Cross-domain JetStream sourcing, how do I set that up?

[tbalbers]

Hi.
** My setup:

• I have a setup with 2x3 NATS nodes set up in a supercluster(cluster01) with JetStream enabled.
• I also have 2 leafnodes(leaf01a and leaf01b) who are standalone, also running JetStream.
• Everything is set up to use TLS, so cluster and supercluster connections are using TLS with certificate-based authentication.
• The leafnodes authenticate to the cluster using their certificates. The system account is not shared between the supercluster and the leafnodes.

All of the above seems to be working fine.

** What I'd like to do:
Now, I'd like to configure this so that:

• I have a JetStream account(cluster01-tx) on the supercluster with subjects replicated to each of the leafnodes.
• On each of the leafnodes, contents from cluster01-tx should end up in leaf01a-cluster01-rx and leaf01b-cluster01-rx.
• I have a JetStream account on each leafnode(leaf01a-tx and leaf01b-tx) with subjects replicated to the supercluster.
• On the supercluster, messages from leaf01a-tx and leaf01b-tx should end up in the account leaf01-rx in the same subjects regardless of which leafnode the messages come from.

** My issues:
I haven't been able to figure out how to do this, in the documentation and examples I've found either this is not done using TLS, og it's done using 'nsc' which I haven't been able to get to work when the NATS server uses TLS(and we also have a requirement to use plaintext(readable) config files for automation, so JWT is probably not going to be acceptable).

I've added some exports and imports on the clusternodeas and leafnodes, but I just end up with errors like:

Error adding service import "$JS.leaf01a.API.CONSUMER.CREATE.tank": service import not authorized

So:
A) Can anyone point me to where I can find some documentation for setting this up? I've been through NATS by example, "jetstream-leaf-nodes-demo" and a bunch of other places without any luck at all.

B) I also find it close to impossible to figure out what all the different messages in the log about adding JetStream domain mapping etc. means. Any pointers to documentation explaining this is much appreciated.

C) Furthermore it's unclear to me if all 3 accounts should exist in the config files on all hosts, clusternodes and leafnodes -and if so how they should be configured.

Thanks in advance.

/tony

[mg1986jp]

Cross-domain JetStream sourcing in a supercluster with leafnodes is a bit tricky. The leafnodes need to be able to reach the JetStream streams in the supercluster, and the way to do this is through account-level imports/exports of the JetStream API subjects.

In your supercluster account config, you need to export the $JS.API.> subjects so the leafnodes can access them. On the leafnode side, import those subjects into the same account. This lets the leafnode's JetStream clients talk to the supercluster's JetStream.

For the sourcing itself, when you create a stream on the leafnode that sources from the supercluster, the stream's sources config needs to reference the stream name as it exists in the supercluster. If the accounts are different, you'll also need to set the external field in the source config with the api prefix pointing to the imported JetStream API subject.

Make sure the leafnode's credentials have permission to subscribe to the subjects being sourced. TLS cert-based auth should work fine here as long as the cert maps to the right user/account on both sides.