Repository navigation
Bugfix/1467 Missing structures after server crash/restart - #2831
Conversation
…uctures Demolishing or decaying a lower shelter re-homes its surviving children (upper shelter, gate, loot) into the parent foundation's freeplaceEntities. The decay vacant-foundation cleanup and the grief-foundation cleanup decided vacancy from the slot maps only, ignoring freeplaceEntities and occupiedRampSlots, so a deck still holding those re-homed structures read as empty and was destroyed after the vacancy timer -- deleting standing shelters/gates and their loot in memory, which then never persisted to the next save (the cause of the recurring 'missing structures after restart' reports). Require freeplaceEntities and ramps to be empty at all three vacancy gates (grief check, expansion-empty test, foundation vacancy gate). Adds regression tests covering survival of re-homed freeplace structures on both the vacant and grief paths, plus a guard that a genuinely empty foundation is still removed.
…orphaning them ConstructionParentEntity.destroy() only re-homed freeplace entities for SHACK types; destroying any other populated parent (e.g. an admin removing a deck/expansion with the weapon remover) left its shelters, gates and loot live in the world dictionaries but unreachable from any saved foundation, so they were dropped on the next save -- the same orphaning class as the decay bug. It now re-homes every child (slotted + freeplace) onto the surviving parent foundation when one exists (e.g. an expansion removed from a live deck), and otherwise promotes loot to world-owned (persisted) and simple structures to world-simple; sub-expansions are destroyed so their own contents are preserved too. ConstructionChildEntity.destroy() now promotes survivors to world-owned when the parent chain is already gone (broken-chain cascade) instead of silently dropping them. Known limitation: _worldSimpleConstruction is not persisted, so simple structures left on a fully-removed top deck still won't survive a restart (full fix needs a save-format change). Adds regression tests for the re-home, world-promotion and broken-chain cases.
…e regenerated world-simple dict _worldSimpleConstruction holds server-generated breakable world props (e.g. workbenches spawned from Z1_taskProps at boot) and is intentionally NOT persisted -- it is regenerated fresh each restart, so persisting it would duplicate world props. Promoting player structures into it (the prior approach, inherited from the SHACK path) neither preserves them nor belongs there. When a whole deck is removed (no surviving parent) structural pieces are now cascade-destroyed while any nested loot is preserved as world-owned (_worldLootableConstruction, which IS persisted). The expansion-removed case still re-homes children onto the surviving deck and persists. Moved the recursive cascade into a module-level cascadeDestroyConstructionChild() in constructionchildentity.ts (imported by constructionparententity) to avoid a circular-import crash: referencing ConstructionParentEntity as a value from constructionchildentity activated a load cycle (ConstructionParentEntity extends ConstructionChildEntity, undefined at module load). Regression tests updated to assert cascade + nested-loot preservation.
Validates the literal bug report: a shelter re-homed onto a foundation's freeplaceEntities (with loot stored inside it) survives the decay vacancy tick AND a serialize -> JSON -> reconstruct round-trip (getConstructionParentSaveData -> JSON.parse(JSON.stringify) -> loadConstructionParentEntity) -- i.e. what a server restart actually exercises. Fails without the Fix 1 decay change (the deck is wiped before the round-trip) and passes with it.
…g an occupied slot Two upper shelters resolving to the same slot on one lower shelter would overwrite each other in occupiedShelterSlots (setSlot wrote unconditionally), orphaning the first upper -- and the loot nested inside it -- so it was unreachable from the foundation slot graph and silently dropped from the next world save, vanishing invisibly on restart while the rest of the base remained. This matches the operator's reported symptom (an upper shelter + its contents gone after a restart), which the earlier decay/destroy commits on this branch do NOT address. setSlot now rejects a different entity on an already-occupied slot (returns false), with a self-id check that keeps reloading the same save idempotent. Placement already gates on occupancy; on load the existing freeplace fallback re-homes the rejected entity instead of dropping it. The guard is in the shared setSlot, so it protects wall/shelter/ramp/expansion slots alike. Adds a red-green regression test.
…l data
LOAD (H1): every Object.values(slotMap) in loadConstructionChildSlots/loadConstructionParentEntity is now guarded with ?? {} and each child loads inside its own try/catch. Previously a foundation save doc missing a slot map (schema drift / older-schema / partial crash-time write) made Object.values(undefined) throw; the per-foundation catch in loadConstructionParentEntities then silently discarded the ENTIRE foundation (every gate/shelter/loot under it), after which deleteMany erased its doc permanently. Now a corrupt child degrades only itself and the base + siblings survive. This is the cleanest match to the reported 'whole base gone after a crash/restart'.
SAVE (H2): each foundation and world-lootable is serialized inside its own try/catch in saveWorld; a throwing foundation is skipped instead of aborting the entire world save (which rolled back and then skipped the final save, losing every base's recent changes). Skipped ids are retained and unioned into deleteMany's $nin keep-set (saveConstructionData / saveWorldFreeplaceConstruction) so a transient serialization error does not permanently delete that foundation's last-good doc.
SHUTDOWN (H3): always attempt one best-effort final save even after a failed periodic save (worldSaveFailed suppresses retry storms, not the final save); a failed periodic save no longer re-enters shutdown when one is already in progress. Adds a load-resilience regression test.
…H5/H6) H4 (expansion load): only force-insert an expansion into a genuinely EMPTY slot (never clobber an already-registered one into an orphan); when the slot is unusable (0/garbled) or occupied, re-home the expansion's children onto the foundation (new WorldDataManager.rehomeChildrenToFoundation) so its shelters/gates/loot stay reachable from the save graph instead of being dropped with the slot-less shell. H5 (ramp load): the ramp/stairs load loop was the only slot loop that discarded setRampSlot's result with no fallback; it now re-homes a rejected ramp to freeplace like the wall/shelter loops. H6 (placement): placeConstructionWall / placeConstructionRamp / placeConstructionStairs / the expansion branch of placeConstructionFoundation now check the set*Slot() result and bail (placementError + return false) instead of inserting the entity into _constructionSimple/_constructionFoundations when the slot was rejected — mirroring placeConstructionShelter/Door and preventing a guard-rejected entity from becoming a live-but-unsaveable orphan. Adds an expansion-re-home regression test.
H11: getLootableConstructionSaveData now serializes the container via getLoadoutContainerSaveData (the canonical primitive-only shape characters/vehicles use, and this field's declared type) instead of embedding the live LoadoutContainer — whose runtime fields (e.g. a stored weapon's reloadTimer) could make the save object non-serializable and abort the entire world save. H12: removeUntransferableFields is now cycle-safe (WeakSet visited-set), so a circular reference can no longer recurse until the stack overflows synchronously inside the save build (which would abort the save). H13: saveWorldFreeplaceConstruction skips the mongo deleteMany($nin:[]) on a transient-empty snapshot so it can't wipe ALL world-lootable (including re-homed player loot); solo still does a full overwrite that correctly reflects an empty world.
After building the construction save data, saveWorld now collects every characterId reachable from the save graph (foundations recursed through all slot maps + freeplace + expansions, plus world-lootable) and logs any construction entity that is live in the world dictionaries but UNREACHABLE — i.e. an orphan that would be silently dropped on the next save. Log-only: it converts any FUTURE orphan-introducing regression into a loud, actionable warning instead of silent shelter/gate loss, without the risk of an auto-re-home. Skipped on a partial save (a foundation failed to serialize and was retained, so its children are legitimately absent this pass). O(n) single pass over data already iterated; cold interval-driven path. Adds a backstop regression test (collectReachableConstructionIds + findSaveGraphOrphans).
The orphan backstop added in 7155501 ran as two synchronous O(N) passes on every save, which on a 150-player / ~50k-entity world is a ~20-60ms event-loop stall. Make it production-safe without weakening detection: - Operator gate + sampling: a new construction.constructionOrphanCheck config flag (default true) decides whether to run, and it only runs on every Nth save (_orphanCheckSaveInterval) so most saves pay nothing. - Fold the reachable-id collection into saveWorld's existing (already yielding, every 100) foundation/world-construction build loops via the new addReachableConstructionIds helper, eliminating the separate synchronous reachable-graph walk. - findSaveGraphOrphans is now async and yields every 2000 scanned entries so the dictionary scan never blocks the tick in one burst. collectReachableConstructionIds is retained (now delegating to addReachableConstructionIds) for the orphan-backstop test. Config flag wired through defaultconfig.yaml, models/config.ts, configmanager.ts, and ConstructionManager following the existing construction-config pattern.
…vation, expansion rehome) A multi-agent adversarial review of the branch confirmed no critical/high/medium issues; these are the low-severity items worth correcting: - H14 orphan-backstop log: the message claimed flagged entities "will be dropped on this save", but the reachable set is a per-foundation snapshot, so an entity placed mid-save can appear transiently while still being reachable via its live parent (it persists on the next save). Reworded to describe snapshot semantics, and capped the listed orphans (first 50 + "...and N more") so a mass-orphan regression cannot build a multi-megabyte log line synchronously on the save path. - H14 comments: corrected two comments that described the reachable-id collection as costing "no extra synchronous pass" - it is an additive recursive walk, hoisted into the (already-yielding) build loops so it shares their yield cadence, not eliminated. - removeUntransferableFields (H12): the added value-is-not-null check incidentally routed null-valued fields into the delete branch (plus "Invalid value type" log spam), whereas they were previously preserved. Restored prior behavior with an explicit null short-circuit; the cycle guard is unchanged. - rehomeChildrenToFoundation (H4): it omitted occupiedExpansionSlots, so a child-parent nesting a sub-expansion (only reachable from corrupt slot data) would orphan that subtree under the deleted shell. Drain expansions recursively with a cycle-guarding seen-set.
…a vacant deck Fix 1 (c81eba6) made the three decay vacancy/grief gates treat a deck as non-vacant whenever freeplaceEntities was non-empty, to stop decay wiping re-homed structures (upper shelters / gates left after a supporting piece is removed). That was too broad: freeplaceEntities also holds plain loot/storage containers, so a single storage box made an otherwise-empty deck permanently immune to vacant-foundation decay and grief cleanup. That is a real regression - a storage container must not keep an abandoned deck alive. Narrow the three gates with a hasStructuralFreeplace() helper that counts only re-homed STRUCTURES (ConstructionChildEntity / ConstructionDoor), not loot (LootableConstructionEntity, which lives on a separate class branch). Now: - a deck holding a re-homed shelter/gate is still spared (Fix 1's real intent); - a deck holding only a loot/storage container decays and despawns with it. Note: #1467's actual reported failure (a single shelter + its loot vanishing from an otherwise-intact base on restart) is the setSlot overwrite-orphan, fixed separately in 4248fef; this decay path is a distinct, latent concern. Added a regression test (a lone loot container does not keep a vacant deck alive); the existing structure-protection tests already use ConstructionChildEntity and remain green.
Two follow-ups from the runtime-detach investigation: 1. setSlot diagnostic (constructionchildentity.ts): when the overwrite guard rejects a different entity from an occupied slot, log the parent, slot, and both entities at error level. This rejection is the exact moment the slot-overwrite orphan (an upper shelter + its loot silently dropped on the next save) WOULD have happened on the pre-fix unguarded code. The investigation proved the loss but could not name the in-game trigger that creates the colliding placement (naive double-placement is OVERLAP-gated). Once deployed, this log captures the real trigger from server logs without reproducing it, and confirms the fix is catching it in the field. It only fires when an overwrite is actually prevented, so it is silent on a healthy server. 2. demolishConstructionEntity (constructionmanager.ts): the undo-placement branch destroyed the entity but had no return, so it fell through and called entity.damage() again on the now-destroyed/detached entity (a double-action). Return after destroy; weapon-item wear is preserved. Unrelated to #1467; a latent wart surfaced by the same investigation.
|
Rewritten player-facing patch notes (hedged — work done, needs further testing) Note: This release contains a large amount of work targeting the long-standing issue of shelters and their stored items disappearing after a server restart. We've identified and fixed a root cause and significantly hardened the construction save/load system against this kind of data loss, but the exact in-game trigger hasn't been fully confirmed yet, so this needs further live testing. Please keep reporting any structures or loot that go missing — diagnostic logging has been added to help pinpoint it. Fixed Issues Addressed the issue where shelters, gates, and the items stored inside them could disappear after a server restart — stacked (upper) shelters in particular should no longer be dropped from the world save (continued reports welcome while we confirm) A deck holding only a storage box or other loose item is no longer treated as occupied and will decay and despawn like any other empty deck |
|
NOTE: This PR has not yet been in-game tested. Serverside simulations have been run in order to try to reproduce this bug, and for testing the "hardening" fixes specifically with the construction system. Further testing is required, as well as careful code review since this PR was written by agents. The first few commits were targeting a false-positive for the #1467 bug, and some things were corrected in further commits. The actual root-cause in-game that caused shelters to become "orphans" is still unknown, however the preventative measures added in this PR should fully address it, prevent it from happening in the future, and document in the console on instances where it could occur again. |
Runs prettier --write over the five files flagged by `prettier --check ./src`. Formatting only; no logic changes. The two test files show large diffs because prettier wraps the long top-level test(...) signature, which re-indents the whole test body one level deeper — pure re-indentation, the assertions are unchanged. Verified: prettier --check passes on all five files; tsc --noEmit clean.
1ac3d0e to
6c0b6bd
Compare
A prior merge left two separate imports of ConstructionChildEntity from ./constructionchildentity, which broke tsc (TS2300) and therefore the npm i postinstall build across every CI matrix job.
Summary
Fixes #1467: players report a shelter — most often an upper shelter — vanishing on a server restart along with all the loot stored inside it, while the foundation, the lower shelter, and the rest of the base remain intact. No decay, no raid, no manual removal; it just disappears after the restart, and it recurs every wipe on fresh data.
This PR fixes the root cause and adds defense-in-depth across the entire construction save/load path so this class of silent data loss cannot recur through any other route.
Root cause
Construction is persisted as one document per top-level foundation, with children (walls, shelters, upper shelters, gates, loot) nested by a recursive serializer that walks each parent's slot maps and freeplace list. This creates a hard invariant:
ConstructionChildEntity.setSlot()wroteoccupiedSlots[slot] = entityunconditionally. When a second entity resolved to a slot already occupied by a different one (an upper shelter is the common real-world case), the incumbent was overwritten out of its parent'soccupiedShelterSlotswhile remaining live in_constructionSimple. It became unreachable from the foundation's save graph and was dropped — with its nested loot — on the next save. The foundation and lower shelter survived because they were still in the graph. This was reproduced end-to-end against the pre-fix code (place → overwrite → save → reload): the orphaned upper and its loot are gone after reload, everything else intact.The fix
setSlot()now refuses to overwrite an occupied slot with a different entity (re-setting the same entity stays idempotent, so reloads are safe):setSlotis the single writer to every slot map, so this is a universal chokepoint:freeplaceEntitiesvia the existing load fallback instead of overwriting — so the structure and its loot both survive rather than one being dropped.Defense-in-depth (the rest of the branch)
Layered so an orphan can't reach the save through any other path:
?? {}+ per-childtry/catch(one corrupt child no longer discards a whole base); the world save isolates per-foundation serialization failures and retains their ids so the prune step can't erase a base whose save momentarily failed; the shutdown save is best-effort; serialization is cycle-safe.setSlotlogs whenever it blocks an overwrite — i.e. exactly when this bug would have occurred — with the parent, slot, and both entities, so the precise in-game trigger can be identified from server logs after deploy.Behavior change (decay)
The decay vacancy/grief checks now distinguish structural freeplace from loot: a deck holding a re-homed shelter/gate is no longer wrongly wiped as "vacant," but a deck holding only a storage/loot container is not kept alive by it and decays normally.
Changes by area
Area | Files | What -- | -- | -- Core fix | entities/constructionchildentity.ts | setSlot overwrite guard; broken-parent-chain handling; cascadeDestroyConstructionChild; overwrite diagnostic Placement | managers/constructionmanager.ts | bail-on-slot-failure; expansion/ramp guards; demolish-undo double-action fix Parent destroy | entities/constructionparententity.ts | preserveChildrenOnDestroy re-homes children to a surviving deck Save/load | managers/worlddatamanager.ts | load ?? {}+try/catch; per-foundation save isolation + retain-ids; expansion/ramp re-home; container/cycle hygiene Save graph | zoneserver.ts | best-effort shutdown save; gated/sampled orphan backstop Decay | managers/decaymanager.ts | structural-vs-loot vacancy gating Config | models/config.ts, managers/configmanager.ts, defaultconfig.yaml | construction.constructionOrphanCheck flag Tests | *.test.ts | regression coverage for every layerTesting
Type-check clean (TS 6.0.3); construction + decay + utils suites green, including new regressions: setSlot overwrite-rejection, missing-slot-map load, unplaceable-expansion re-home, orphan backstop, loot-only deck decays, and a full save→reload round-trip with re-homed structures + nested loot.
Important deployment note
The
setSlotguard is not ondevtoday, so live servers keep losing structures every wipe until this merges. Merging this PR is what ships the fix.Known / follow-up (transparent)
The outcome (overwrite → orphan → drop) is reproduced and fixed. The exact in-game event that creates the slot collision on an intact base is not yet named — naive double-placement is already blocked by the overlap gate, so the trigger lives in the client→placement packet path. The added
Summary Fixes [#1467](https://github.com//issues/1467): players report a shelter — most often an upper shelter — vanishing on a server restart along with all the loot stored inside it, while the foundation, the lower shelter, and the rest of the base remain intact. No decay, no raid, no manual removal; it just disappears after the restart, and it recurs every wipe on fresh data.setSlotdiagnostic is designed to capture it from production logs within one wipe cycle (and confirm the fix is catching it). One unrelated latent wart was also fixed:demolishConstructionEntityfell through after an undo-placementdestroy()and re-hit the removed entity.This PR fixes the root cause and adds defense-in-depth across the entire construction save/load path so this class of silent data loss cannot recur through any other route.
Root cause
Construction is persisted as one document per top-level foundation, with children (walls, shelters, upper shelters, gates, loot) nested by a recursive serializer that walks each parent's slot maps and freeplace list. This creates a hard invariant:
A construction entity is saved only if it is reachable from a top-level foundation's in-memory slot/freeplace graph at save time. Anything live in memory but unreachable is an orphan, silently dropped from the next save — and gone on the next restart.
ConstructionChildEntity.setSlot() wrote occupiedSlots[slot] = entity unconditionally. When a second entity resolved to a slot already occupied by a different one (an upper shelter is the common real-world case), the incumbent was overwritten out of its parent's occupiedShelterSlots while remaining live in _constructionSimple. It became unreachable from the foundation's save graph and was dropped — with its nested loot — on the next save. The foundation and lower shelter survived because they were still in the graph. This was reproduced end-to-end against the pre-fix code (place → overwrite → save → reload): the orphaned upper and its loot are gone after reload, everything else intact.
The fix
setSlot() now refuses to overwrite an occupied slot with a different entity (re-setting the same entity stays idempotent, so reloads are safe):
const existing = occupiedSlots[slot];
if (existing && existing.characterId !== entity.characterId) {
return false;
}
setSlot is the single writer to every slot map, so this is a universal chokepoint:
At placement, a conflicting placement is rejected cleanly instead of silently orphaning the incumbent.
At load, a conflicting saved entry is re-homed to the parent's freeplaceEntities via the existing load fallback instead of overwriting — so the structure and its loot both survive rather than one being dropped.
Defense-in-depth (the rest of the branch)
Layered so an orphan can't reach the save through any other path:
Prevent: placement callers now bail (and clean up) on a failed slot assignment instead of leaving a half-placed entity; expansion/ramp load conflicts re-home rather than clobber.
Tolerate: load loops use ?? {} + per-child try/catch (one corrupt child no longer discards a whole base); the world save isolates per-foundation serialization failures and retains their ids so the prune step can't erase a base whose save momentarily failed; the shutdown save is best-effort; serialization is cycle-safe.
Detect: an operator-gated, sampled, yielding save-time backstop logs any construction that is live but unreachable from the save graph, turning a future regression into a loud warning instead of silent loss.
Diagnostic: setSlot logs whenever it blocks an overwrite — i.e. exactly when this bug would have occurred — with the parent, slot, and both entities, so the precise in-game trigger can be identified from server logs after deploy.
Behavior change (decay)
The decay vacancy/grief checks now distinguish structural freeplace from loot: a deck holding a re-homed shelter/gate is no longer wrongly wiped as "vacant," but a deck holding only a storage/loot container is not kept alive by it and decays normally.
Changes by area
Area Files What
Core fix entities/constructionchildentity.ts setSlot overwrite guard; broken-parent-chain handling; cascadeDestroyConstructionChild; overwrite diagnostic
Placement managers/constructionmanager.ts bail-on-slot-failure; expansion/ramp guards; demolish-undo double-action fix
Parent destroy entities/constructionparententity.ts preserveChildrenOnDestroy re-homes children to a surviving deck
Save/load managers/worlddatamanager.ts load ?? {}+try/catch; per-foundation save isolation + retain-ids; expansion/ramp re-home; container/cycle hygiene
Save graph zoneserver.ts best-effort shutdown save; gated/sampled orphan backstop
Decay managers/decaymanager.ts structural-vs-loot vacancy gating
Config models/config.ts, managers/configmanager.ts, defaultconfig.yaml construction.constructionOrphanCheck flag
Tests *.test.ts regression coverage for every layer
Testing
Type-check clean (TS 6.0.3); construction + decay + utils suites green, including new regressions: setSlot overwrite-rejection, missing-slot-map load, unplaceable-expansion re-home, orphan backstop, loot-only deck decays, and a full save→reload round-trip with re-homed structures + nested loot.
Important deployment note
The setSlot guard is not on dev today, so live servers keep losing structures every wipe until this merges. Merging this PR is what ships the fix.
Known / follow-up (transparent)
The outcome (overwrite → orphan → drop) is reproduced and fixed. The exact in-game event that creates the slot collision on an intact base is not yet named — naive double-placement is already blocked by the overlap gate, so the trigger lives in the client→placement packet path. The added setSlot diagnostic is designed to capture it from production logs within one wipe cycle (and confirm the fix is catching it). One unrelated latent wart was also fixed: demolishConstructionEntity fell through after an undo-placement destroy() and re-hit the removed entity.