Repository navigation
Killed data fsck unit (Result=signal) is reported as data_filesystem_check_error #7374
Description
Describe the issue you are experiencing
After a power loss, the system was marked unhealthy with data_filesystem_check_error ("Data partition filesystem check failed ... may affect configuration, apps, or backups"). The data filesystem is not damaged. The check never finished: systemd stopped systemd-fsck@dev-disk-by\x2dlabel-hassos\x2ddata.service with SIGTERM about 130 ms after e2fsck started replaying the ext4 journal.
09:29:51.575950 systemd[1]: Starting File System Check on /dev/disk/by-label/hassos-data...
09:29:51.601526 systemd-fsck[341]: e2fsck 1.47.2 (1-Jan-2025)
09:29:51.606423 systemd-fsck[341]: hassos-data: recovering journal
09:29:51.736013 systemd[1]: systemd-fsck@dev-disk-by\x2dlabel-hassos\x2ddata.service: Main process exited, code=killed, status=15/TERM
09:29:51.738423 systemd[1]: systemd-fsck@dev-disk-by\x2dlabel-hassos\x2ddata.service: Failed with result 'signal'.
09:29:51.738554 systemd[1]: Stopped File System Check on /dev/disk/by-label/hassos-data.
09:29:52 kernel: EXT4-fs (sda8): mounted filesystem b0bf7487-... r/w with ordered data mode.
systemctl show on the unit: Result=signal, ExecMainCode=2, ExecMainStatus=15. The "Stopped" line points to a stop job, not to e2fsck failing. The likely cause is the open systemd race systemd/systemd#43695: the journal replay closes a writable fd on the partition, udev synthesizes a change event, blkid briefly returns no filesystem while the replay runs, the /dev/disk/by-label/hassos-data link disappears, and since systemd-fsck@.service has BindsTo=%i.device, the running check is stopped. The small overlay partition replayed its journal in the same boot and finished in time; the 64 GB data partition did not.
To confirm the filesystem was fine, I ran e2fsck -f -y on a block-level snapshot of the VM disk: it replayed the journal and only fixed the free block and inode summary counts, with no structural errors. The superblock records no errors, and the recorder database passes PRAGMA quick_check. The three earlier (clean) boots all logged hassos-data: clean.
The check in supervisor/resolution/checks/systemd_unit_failure.py treats any failed state of the data fsck unit as a filesystem check failure. A unit that was killed or stopped before it could report has not found a problem.
Suggested fix: read the unit's Result property (org.freedesktop.systemd1.Service.Result). Only exit-code means the checker ran and returned a failure (uncorrectable errors or reboot required, given how systemd-fsck maps exit codes). For signal, timeout or similar, the check is inconclusive: report it as an issue such as "filesystem check did not complete, reboot to run it again" rather than a critical unhealthy state that says the data may be corrupted. The same applies to OS_FSCK_UNITS.
What type of installation are you running?
Home Assistant OS
Which operating system are you running on?
Home Assistant Operating System
Steps to reproduce the issue
- Run HAOS with a reasonably large data partition (64 GB here, Proxmox VM, virtio-scsi on LVM-thin).
- Cut power while it is writing, so the data partition's ext4 journal needs replaying.
- Boot: if the udev change event lands while e2fsck is still replaying, the data fsck unit is stopped with SIGTERM, and Supervisor marks the system unhealthy with
data_filesystem_check_error.
It depends on timing, so it will not happen on every unclean shutdown.
Anything in the Supervisor logs that might be useful for us?
2026-10-14 09:30:01.638 ERROR (MainThread) [supervisor.resolution.checks.systemd_unit_failure] Data filesystem check failed: systemd-fsck@dev-disk-by\x2dlabel-hassos\x2ddata.service
2026-10-14 09:30:01.654 INFO (MainThread) [supervisor.jobs] 'ResolutionFixup.run_autofix' blocked from execution, system is not healthy - data_filesystem_check_error
2026-10-14 09:30:01.654 CRITICAL (MainThread) [supervisor.core] System is running in an unhealthy state and needs manual intervention!System information
- Installation type: Home Assistant OS
- Core: 2026.10.1
- Supervisor: 2026.09.3
- Operating System: Home Assistant OS 18.3 (kernel 6.18.52-haos, systemd 256.17)
- Board: qemux86-64 (Proxmox VE 9, q35, virtio-scsi-single, data disk on LVM-thin)
- Docker: 29.7.2
- Channel: stable
Additional information
A related gap: with fsck.repair=yes, systemd-fsck exits successfully when e2fsck corrected errors (exit status 1), so this check never surfaces corruption that was found and repaired at boot. Only uncorrectable errors fail the unit. So today the warning can fire when nothing was checked, and stay silent when something was repaired. A repaired filesystem could be worth an issue too, but the obvious signal (/run/systemd/quotacheck, touched on exit status 1) is also set by routine journal replays that only fix summary counts, so it would need care.
This report was researched with an AI assistant (Claude Code) and reviewed before posting.
Activity
agners commentedon Oct 15, 2026
Interesting, can you check why it got killed? E.g. anything in the Host logs about it?
mayerwin commentedon Oct 15, 2026
Thanks for looking. Unfortunately the host journal has no line that names the reason. These are all the lines for the data partition on that boot, plus the units around it. The rest of that window is Bluetooth firmware loading and udev CPU rules.
What this rules out:
Conflicts=on the unit. There was no OOM kill and no timeout either: the result issignal, nottimeout, and the unit ran for about 160 ms.mnt-data.mountonly hasWants=on the fsck unit, andhaos-data-disk-detach.serviceonly hasAfter=.That leaves
BindsTo=%i.deviceondev-disk-by\x2dlabel-hassos\x2ddata.deviceas the only dependency that passes a stop on to the unit. At default log levels, neither PID 1 nor udevd logs a device unit going from plugged to dead and back ("Found device" is only printed when a start job completes), so the journal can't confirm it directly. It does match systemd/systemd#43695:changeevent.The link was back a second later, when the data partition mounted normally.
agners commentedon Oct 15, 2026
Thanks for the thorough analysis, that matches what I see in systemd/systemd#43695 as well. The transient bad superblock checksum during journal replay plus the udev re-probe explains the
BindsTo=stop nicely.I don't think it is worth teaching Supervisor to tell a killed check apart from a failed one, though. In both cases the filesystem check did not complete, and the action for the user is the same: reboot so it runs again. The unhealthy state is not persisted, so after a boot where the fsck unit finishes it goes away. I assume that is what happened on your side? The wording could be a bit less alarming, but a separate issue type with its own D-Bus lookup is a lot of machinery for a softer message.
The proper fix belongs in systemd (or e2fsprogs), and there is a draft PR upstream. Our telemetry shows this is rare at the moment, so I would rather wait for that to land than carry a patch in HAOS. If it turns out to hit more installations we can revisit backporting it.
On the
fsck.repair=yesgap: agreed that would be nice to surface, but as you note there is no clean signal for "actually repaired something" versus a routine replay, so I'd leave that as is for now.This reply was drafted with an AI assistant (Claude Code), reviewed by me and reflects my view.