Batch re-syncing after rapid node restarts

[gacevicljubisa]

Description

After restarting my Bee node multiple times with short delays between restarts, I encountered the following issues:

• The node started re-syncing batches from the beginning, even though they were previously synced.

This behavior needs further investigation to understand why rapid restarts cause batch re-syncing.

Steps to Reproduce

1. Restart a Bee node multiple times with short delays between restarts.
2. Observe if batches re-sync from the beginning and if data becomes corrupted.

Expected Behavior

• Previously synced batches should not re-sync unnecessarily.
• Data should remain intact and accessible after restarts.

Additional Context

This issue requires further investigation to identify the root cause and prevent re-syncing and investigate if there is any data loss.

[acud]

needs further investigation. please try to reproduce the corruption using a unit test. ideally i'd try to use an llm to build the scenario and just rinse and repeat, trying to corrupt the database.

[sbackend123]

Investigation results

• The batch service uses a dirty flag in the state store.
• TransactionStart() sets dirty=true and buffers chain state in memory; TransactionEnd() commits chain state and clears the flag.
• The postage listener wraps each RPC log page in TransactionStart → process events → TransactionEnd (pkg/postage/listener/listener.go).
• If the process dies after TransactionStart and before TransactionEnd, the dirty flag can remain. On the next start, Start() detects it, calls storer.Reset(), and the node re-syncs chain postage data from scratch (warnings: “dirty shutdown detected, resetting batch store” and “batch store has been reset…”).
• Partial writes: batch rows can be written during the “transaction” while chain state is only committed at TransactionEnd, so a crash can leave an inconsistent mix—hence the conservative full reset.

Attempts to reproduce:

• Debug logging: Added debug logs in the listener and batch service (block ranges, log counts, processEvents steps, TransactionStart / TransactionEnd, batch service start, etc.) so we can see that the code under investigation actually runs.
• Script: resync.sh starts the node and waits until the log matches the “dirty flag set, processing events” line with "count"=>0 (i.e. after TransactionStart when there is at least one event—otherwise TransactionEnd happens almost immediately and the window is too narrow). On match it sends SIGKILL right away. Maximum wait for the trigger per attempt is 15 minutes.
• Outcome: After trying different timings, configs (including skip-postage-snapshot), and scenarios, we could not reproduce the issue on a real node.

Look at branch fix/resync-after-start. Before running resync.sh script make sure it is in same dir as your bee binary and bee config