ci: Add support for linux-riscv64

[luhenry]

Description

This is currently an experiment. Happy to get reviews as always.

Add GitHub Actions CI on linux-riscv64 using RISE RISC-V Runners.

Relates to #1919
Depends on k0sproject/image-builder#253 #7459

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation update

How Has This Been Tested?

• Manual test
• Auto test added

Checklist

• My code follows the style guidelines of this project
• My commit messages are signed-off
• I have performed a self-review of my code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes
• Any dependent changes have been merged and published in downstream modules
• I have checked my code and corrected any misspellings

[luhenry]

Currently blocked on missing https://github.com/anchore/syft support for linux-riscv64. I'll submit a PR upstream and update here accordingly. EDIT: work in progress at anchore/syft#4757, I disabled it here for now for riscv64 specifically.

[twz123]

Nice! IIUC we need to add the RISE app to our GitHub org, right? There's also https://www.riscvrunners.com/. Do you know the differences?

[luhenry]

Nice! IIUC we need to add the RISE app to our GitHub org, right? There's also https://www.riscvrunners.com/. Do you know the differences?

Yes, you’d need to add https://github.com/apps/rise-risc-v-runners (only require permissions on “self-hosted runners read+write”).

On https://www.riscvrunners.com/, the main difference is the RISE ones are free (and will stay free, RISE is not a business) and you have access to the whole machine. I’m not intimately familiar with the base image they are using (which packages, what OS, etc) but I’ll look more into it.

[luhenry]

Trying to add the proper image dependencies at k0sproject/image-builder@main...luhenry:k0s-image-builder:main

[twz123]

Trying to add the proper image dependencies at k0sproject/image-builder@main...luhenry:k0s-image-builder:main

I need to follow up on k0sproject/image-builder#256. I don't think we can generally enable RISC-V builds for everything.

[luhenry]

Trying to add the proper image dependencies at k0sproject/image-builder@main...luhenry:k0s-image-builder:main

I need to follow up on k0sproject/image-builder#256. I don't think we can generally enable RISC-V builds for everything.

Sounds good. Do you know if there will be other images that will be a required dependency?

[luhenry]

I'm brute-forcing my way through figuring out which images/versions are working: https://github.com/luhenry/k0s-image-builder/actions/runs/24236376271

Good news is quite a few are already working, so adding them is a simple "adding a new version" away. Once that workflow finished, I'll look through all the failures and categorize them, that should give us a good todo list, and list of things to exclude/ignore as well

[twz123]

Sounds good. Do you know if there will be other images that will be a required dependency?

No. In fact, Calico is not a requirement. By default, k0s will use kube-router, which is already available for RISC-V. You won't be able to use NLLB (another optional feature), as this is would require envoy, and we didn't have the time to try to make it compile for RISC-V. But I'm currently looking into adding support for Traefik (#7405), mainly for Windows, but Traefik supports RISC-V and ARMv7 out of the box, so this might be a viable alternative to make NLLB usable on RISC-V, as well.

[luhenry]

Recent development: BIRD has just added (like 3 days ago) builds for riscv64: https://gitlab.nic.cz/labs/bird/-/commit/0aca5b325fb6606d620ef349d70892105e1242c3. I ran into that issue in the build for calico-node at least.

In fact, Calico is not a requirement

Good, then at least it's not a blocker. It should get resolved soon though. I never like diverging from how it's done on linux-amd64 and linux-arm64 because I like things to "just work". But gotta be pragmatic here 😅

[luhenry]

Also, another finding: etcd is broken on all platforms, since bitnami/etcd seems to have no more tags (?). That seems broken not just for linux-riscv64 but for all other platforms as well.

[twz123]

Also, another finding: etcd is broken on all platforms, since bitnami/etcd seems to have no more tags (?). That seems broken not just for linux-riscv64 but for all other platforms as well.

Not all the images in the image-builder repo are used by k0s itself. There's also some utility things, like the GH Actions runners which we use to build k0s for ARMv7. Not sure if the etcd stuff is used anywhere, still.

Pretty sure that envoy is the last image that's directly used by k0s which is not available for RISC-V.

[twz123]

Recent development: BIRD has just added (like 3 days ago) builds for riscv64: https://gitlab.nic.cz/labs/bird/-/commit/0aca5b325fb6606d620ef349d70892105e1242c3. I ran into that issue in the build for calico-node at least.

We started to build BIRD from source in the Calico Dockerfiles recently, so this should be fine.

[luhenry]

Triggering with v3.31.4-2 from k0sproject/image-builder#256, running at https://github.com/luhenry/k0s/actions/runs/24244825789

[luhenry]

And for the missing quay.io/k0sproject/pushgateway-ttl:1.4.0-k0s.0 image: k0sproject/pushgateway-ttl-builder#5

[luhenry]

Following k0sproject/pushgateway-ttl-builder#5 (comment), I've cherry-picked your commit to mark Envoy as not-supported.

[luhenry]

https://github.com/replicatedhq/troubleshoot is missing on riscv64 now for:

• check-basic :: riscv64 / Test
• check-network-conformance-calico :: riscv64 / Test
• check-network-conformance-kuberouter :: riscv64 / Test
• check-airgap :: riscv64 / Test

The simplest solution is to simply not run the smoketests-linux-riscv64 for now. I'll submit a PR upstream to try to get them to enable and release linux-riscv64 as well.

I've also verified that k0sproject/image-builder#253 is working: I rebased it and used the resulting image at https://github.com/luhenry/k0s/actions/runs/24254357499/job/70824468191. The Build :: Airgap image bundle (riscv64) / linux-riscv64 is passing.

[luhenry]

Opened a PR on replicatedhq/troubleshoot#2010 for linux-riscv64

[luhenry]

I got some progress in enabling the test suite but hitting the next blocker.

On the overlay-on-overlay, I fixed it by using an empty volume for the pod at /var/lib/k0s (see [1]). Because a k8s pod's volumne is a direct mount from the host's filesystem into the container, and not an overlayfs, the /var/lib/k0s folder inside the pod is a ext4 mount [2].

The next issue I ran into was the DNS from kube-proxy inside the k0s cluster was getting confused because the CIDR was the same between the k8s's pod and the k0s running inside the pod. The solution was simply to use the hostNetwork for the pods [3]. This actually makes more sense since the pod should not have access to anything internal to the k8s cluster.

Finally, the issue I'm not running into has to do with kube-router trying to setup a firewall which depends on the ip_set_hash_net kernel module. However, that module is not available on the Scaleway EM-RV1 out-of-the-box (even as a module). So the solution would be to build the kernel module so that it can be loaded. I'm currently working on that.

As a side note, we are also working with a Kernel contributor to upstream any patches necessary to mainline Linux to support the TH1520 (the SoC used by the EM-RV1) so we can go to mainline kernel instead of an older kernel version.

Overall, it's making good progress and I'm hoping to get things working in next weeks. Good news is nearly nothing has to change on k0s itself, and all in the infrastructure.

[2]:

$> kubectl exec -it rise-riscv-runner-9572x75pv -- findmnt -R
TARGET SOURCE FSTYPE  OPTIONS
/      overlay
              overlay rw,relatime,lowerdir=/var/lib/containerd/io.containerd.sna
|-/var/lib/k0s
|      /dev/mmcblk0p3[/var/lib/kubelet/pods/a8bffb66-a287-4d9f-a4bf-7cb8c30db806/volumes/kubernetes.io~empty-dir/k0s]
|             ext4    rw,relatime,errors=remount-ro

[twz123]

Good news! CNCF approved to add the RISE RISC-V runners to the k0sproject org 🚀

@luhenry could you rework this PR in a way that it doesn't trigger for every PR? Instead, I think it's best to start with a nightly job that also has a workflow_dispatch, similar to what we have for the ostests.

If the integration tests aren't working yet, we could add this as an optional parameter for workflow_dispatch (again, in a way similar to what we're doing for the ostests, this time in ostests-matrix.yaml). This way, we can execute and iterate on the integration tests without needing to change the workflows.

[luhenry]

I split the unittests in a separate workflow_call file, similar to build-k0s and smoketests. I added a go-nightly.yml which only runs the build-k0s, unittests, build-airgap-image-bundle and smoketests-linux-riscv64 (disabled for now [1][2]) for linux-riscv64.

I'm testing it at:

• go.yml: https://github.com/luhenry/k0s/actions/runs/25448582937
• go-nightly.yml:
  • w/o smoketests: https://github.com/luhenry/k0s/actions/runs/25454194134
  • w/ smoketests: https://github.com/luhenry/k0s/actions/runs/25454212706

[luhenry]

For the latest on the issue with kube-router, I met with my contact at Scaleway today, and they are going to share the sources for the Linux Kernel they have on the Scaleway EM-RV1. I'll build the necessary modules from it and add them to all the machines of the pool. I'll keep you posted as soon as I have more things to show.

@luhenry could you rework this PR in a way that it doesn't trigger for every PR? Instead, I think it's best to start with a nightly job that also has a workflow_dispatch, similar to what we have for the ostests.

If the integration tests aren't working yet, we could add this as an optional parameter for workflow_dispatch (again, in a way similar to what we're doing for the ostests, this time in ostests-matrix.yaml). This way, we can execute and iterate on the integration tests without needing to change the workflows.

@twz123 that should be all done.

[twz123]

If you want, you can split out 6fd5939 into a separate PR, so we can merge that already.

[luhenry]

If you want, you can split out 6fd5939 into a separate PR, so we can merge that already.

#7591

I'll work on the rest of the feedback tomorrow

[twz123]

If you want, you can split out 6fd5939 into a separate PR, so we can merge that already.

#7591

I'll work on the rest of the feedback tomorrow

I meant the split-off of the unit tests into their own callable workflow 😅

[luhenry]

If you want, you can split out 6fd5939 into a separate PR, so we can merge that already.

#7591
I'll work on the rest of the feedback tomorrow

I meant the split-off of the unit tests into their own callable workflow 😅

Ah sorry, my bad, will do that first thing tomorrow!

[luhenry]

@twz123 I split it out as you requested.

Also, I got check-basic to work 🎉 I updated the underlying workers with the right kernel modules and it works. I'll check other test suites now.

UPDATE: check-basic 1 and check-airgap 2 are working. The check-network-conformance-calico 3 and check-network-conformance-kuberouter 4 are not, but only because https://hub.docker.com/r/sonobuoy/sonobuoy is not available on riscv64. I'll try submitting a PR this week

UPDATE 2: The PR for sonobuoy: vmware-tanzu/sonobuoy#2049

[luhenry]

@twz123 anything else I should do to get this merged? Thanks 🙏

[luhenry]

I verified a build works: https://github.com/luhenry/k0s/actions/runs/26634638919

[luhenry]

Enabling the basic smoketests as I've verified it works at https://github.com/luhenry/k0s/actions/runs/26634649027

[luhenry]

Also enabling airgap as it works: https://github.com/luhenry/k0s/actions/runs/26709975774/job/78719293615

[twz123]

Thanks for pushing this forward!

[twz123]

The changes here are a bit hacky, but I don't have a better proposal right now that wouldn't require lots of work. So just let's live with it for now...

[luhenry]

It's been succeeding for past few days:

• https://github.com/k0sproject/k0s/actions/runs/28001505151
• https://github.com/k0sproject/k0s/actions/runs/27928956848