Skip to content

sharky corruption after host crash #4737

Description

@ldeffenb

Context

v2.1.0 on sepolia testnet swarm

Summary

I had 13 nodes running on a single host, 2 on an internal NVME SSD and the other 11 on a USB-3-connected SATA SSD. Due to reasons outside of swarm, this host crashed without warning. On powering up, I discovered that all 11 nodes on the external SSD had sharky corruptions as indicated by db validate. The 2 nodes on the internal NVME SSD validated cleanly.

It is also worth noting that this host crashed while it was very active pushing a new OSM hybrid-redundancy dataset into the testnet swarm. The nodes were definitely NOT idle when the crash happened.

Expected behavior

I would hope that recovery from a system crash would not cause a corrupted sharky. But I can almost understand it with a USB-connected drive.

Actual behavior

Verification results from multiple nodes on that single host.

"time"="2024-07-26 13:22:49.541998" "level"="info" "logger"="node" "msg"="validation finished" "duration"="40m1.635440786s" "invalid"=2 "soc"=1319448 "total"=2137804
"time"="2024-07-26 13:23:33.389734" "level"="info" "logger"="node" "msg"="validation finished" "duration"="40m29.69367306s" "invalid"=41 "soc"=1321454 "total"=2140554
"time"="2024-07-26 13:30:12.261303" "level"="info" "logger"="node" "msg"="validation finished" "duration"="49m38.373076047s" "invalid"=228 "soc"=4198475 "total"=6213034
"time"="2024-07-26 13:24:33.398450" "level"="info" "logger"="node" "msg"="validation finished" "duration"="41m12.410722607s" "invalid"=148 "soc"=1336141 "total"=2159790
"time"="2024-07-26 13:24:23.690402" "level"="info" "logger"="node" "msg"="validation finished" "duration"="40m45.640637335s" "invalid"=22 "soc"=1329395 "total"=2150760
"time"="2024-07-26 13:24:28.904050" "level"="info" "logger"="node" "msg"="validation finished" "duration"="40m36.74342581s" "invalid"=47 "soc"=1319130 "total"=2137422
"time"="2024-07-26 13:24:36.198084" "level"="info" "logger"="node" "msg"="validation finished" "duration"="40m24.78211206s" "invalid"=31 "soc"=1319296 "total"=2137618
"time"="2024-07-26 13:24:15.713920" "level"="info" "logger"="node" "msg"="validation finished" "duration"="39m40.56859582s" "invalid"=8 "soc"=1319289 "total"=2137613
"time"="2024-07-26 13:20:43.081423" "level"="info" "logger"="node" "msg"="validation finished" "duration"="35m53.235082068s" "invalid"=88 "soc"=1325899 "total"=2146762
"time"="2024-07-26 13:24:59.632308" "level"="info" "logger"="node" "msg"="validation finished" "duration"="40m5.682027361s" "invalid"=10 "soc"=1329492 "total"=2151330
"time"="2024-07-26 13:25:11.043347" "level"="info" "logger"="node" "msg"="validation finished" "duration"="40m12.737092877s" "invalid"=5 "soc"=1366998 "total"=2201914

Steps to reproduce

I guess if you have a very active node and crash the host with the data-dir on an external USB drive, you may be able to duplicate the corruption.

Possible solution

Don't run with data-dir on an external drive?

At least it may be worth noting somewhere in the documentation that running on external drives may be at a risk for data corruption.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions