/usr/local from persistent partition is shadowed when sysext merges

[supershal]

I am installing kubernetes binaries using a systemd sysext. After enabling the sysext, the resulted files in /usr/local does not include files form persistent source. /dev/disk/by-label/COS_PERSISTENT

After sysext merge, all files under /usr/local/etc/ from the persistent are not available in the final image. This prevents containerd to start any containers since it cant find /etc/hosts.

The image has following.

> findmnt /usr/local
TARGET     SOURCE                            FSTYPE OPTIONS
/usr/local /dev/disk/by-label/COS_PERSISTENT ext4   rw,relatime,stripe=256

> ls /usr/local
bin  cloud-config  etc  lib  lost+found  share

>ls /usr/local/etc
hostname  hosts  machine-id

• Install Kubernetes sysext

> systemd-sysext merge
Using extensions 'kubernetes-1.35.0-amd64'.
Merged extensions into '/usr'.
Merged extensions into '/opt'.

• /usr/local after sysext merged

> ls /usr/local
bin  etc  games  include  lib  man  sbin  share  src

>ls -l /usr/local/etc/
total 0

It completely ignores content of the persistent partitions and exposes the base OS's files. games, src, etc.

I found some related issues filed in the systemd
Issue reported: systemd/systemd#31922
supposed fix: systemd/systemd#34381
How to prevent this when working with sysext?

[supershal]

I am running ubuntu-24.04 which comes with systemd 255. The fix seems to be available systemd > 257
Is there any workaround suggested for older systemd?

[Itxaka]

Hello @supershal

this seems to be an issue with the shell. We dont allow sysext to configure the /usr directly because of the read only/ shadow that it does. We have a specific set of dirs that are used for sysext and those are loaded both by the service and a profile env var. This is explained here

Depending on the shell it should autoload them automatically and be set so when you use sysext it sets the proper dirs without overriding our persistent ones:

https://github.com/kairos-io/kairos-init/blob/main/pkg/bundled/cloudconfigs/99_sysext.yaml#L108

In this case if you do an env|grep -i HIERARCHIES you should see an entry. Otherwise something is misconfigured or your shell didnt login so it didnt load the profile values. If you cant loging you can always source /etc/profile.d/systemd-sysext.sh and try again, but the services already have those values set so they should be enabled automatically.

Al;so you can do management of the sysext with kairos-agent which also includes the hierarchies automatically.