Skip to content

Commit f75d38d

Browse files
committed
fix: replace the tail watchdog with a supervisor that revives dead tails
The watchdog inferred liveness from a bare heartbeat timestamp, so a tail that exited cleanly (journal permanently lost, rescan budget spent, panic) looked identical to one wedged in a stuck disk read: it error-logged and cancelled I/O on the dead thread every 30 seconds forever, while nothing ever brought the volume back. Each tail now publishes Beat/Busy/Retired in one atomic word, with Retired published by a drop guard so every exit path, panics included, reports it. A supervisor owns the tails: stale beats get their blocked I/O cancelled (wedged rebuilds too, on a larger bound), retired tails are revived on a bounded budget (reopen, respawn, rescan first), and a volume that keeps dying gets one error and an honest phase instead of eternal alarms. The budget refunds only after ten minutes of fresh Live beats. Also fixed along the way: - rescan captured its journal cursor after the MFT enumeration, silently losing changes that landed mid-rebuild; it now captures first, matching bootstrap, and a journal replaced mid-rebuild fails loudly instead - teardown is wedge-tolerant end to end: flag every tail, cancel-and-poll to a deadline, abandon stragglers loudly; blocking revival I/O runs on worker threads so one sick volume cannot stall supervision of the rest - a failed thread-handle duplication no longer leaks a live unsupervised writer, and a tail marking its volume Failed now logs the exit instead of dying silently
1 parent 5290d75 commit f75d38d

10 files changed

Lines changed: 1237 additions & 156 deletions

File tree

README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@ goz indexes every filename on your fixed NTFS volumes and answers substring quer
1313

1414
- **Fast.** 1.8x to 6.1x quicker than Everything across every query in our [benchmarks](docs/BENCHMARKS.md), on the same machine with the same result sets.
1515
- **Light.** ~215 MB resident under heavy querying and ~12 MB idle, against a ~3.97M-entry index.
16-
- **Live.** Tails the NTFS USN journal, so the index stays in sync as files change, including hard-link renames that Everything 1.4 misses.
16+
- **Live.** Tails the NTFS USN journal, so the index stays in sync as files change, including hard-link renames that Everything 1.4 misses. A tail thread that wedges on a sick disk or dies outright is cancelled or revived automatically.
1717
- **Scriptable.** Native `--json` output and a subset of `es.exe`-compatible flags with matching exit codes, so it drops into existing Everything workflows.
1818
- **Honest about state.** Every volume reports whether it is provably in sync, rescanning, or offline. It never serves stale results silently.
1919

crates/goz-core/src/usn/cursor.rs

Lines changed: 9 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -56,11 +56,11 @@ pub enum Resync {
5656
}
5757

5858
/// Why a full rescan was decided. [`validate_cursor`] produces only
59-
/// `JournalIdChanged` / `CursorPurged` / `RestampDiscontinuity`; `ParserAnomaly` and
60-
/// `EntryDeletedError` are mapped by the daemon from runtime conditions.
61-
/// `JournalNotActive`, `JournalDeleteInProgress`, and `VolumeReattached` are
62-
/// reserved for causes the daemon does not yet map. All live here so every
63-
/// rescan cause shares one telemetry vocabulary.
59+
/// `JournalIdChanged` / `CursorPurged` / `RestampDiscontinuity`; `ParserAnomaly`,
60+
/// `EntryDeletedError`, and `TailRevived` are mapped by the daemon from runtime
61+
/// conditions. `JournalNotActive`, `JournalDeleteInProgress`, and
62+
/// `VolumeReattached` are reserved for causes the daemon does not yet map. All
63+
/// live here so every rescan cause shares one telemetry vocabulary.
6464
#[derive(Clone, Debug, PartialEq, Eq)]
6565
pub enum RescanReason {
6666
/// Saved `UsnJournalID` differs from the live journal's: the journal
@@ -85,6 +85,10 @@ pub enum RescanReason {
8585
/// A corrupt buffer or cycle was detected while parsing/applying; we
8686
/// cannot prove what was missed (daemon-mapped).
8787
ParserAnomaly,
88+
/// The volume's tail thread exited and its supervisor revived it; the
89+
/// volume went untailed in between, so nothing proves the index is still
90+
/// in sync (daemon-mapped).
91+
TailRevived,
8892
/// The volume went away and came back; its journal may have advanced on
8993
/// another machine.
9094
VolumeReattached,

crates/goz-daemon/src/main.rs

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,12 +16,16 @@ static GLOBAL: mimalloc::MiMalloc = mimalloc::MiMalloc;
1616
#[cfg(windows)]
1717
mod bootstrap;
1818
#[cfg(windows)]
19+
mod pulse;
20+
#[cfg(windows)]
1921
mod run;
2022
#[cfg(windows)]
2123
mod server;
2224
#[cfg(windows)]
2325
mod service;
2426
#[cfg(windows)]
27+
mod supervisor;
28+
#[cfg(windows)]
2529
mod tail;
2630
#[cfg(windows)]
2731
mod volume_state;

crates/goz-daemon/src/pulse.rs

Lines changed: 178 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,178 @@
1+
//! A tail thread's self-reported liveness, packed into one atomic word.
2+
//!
3+
//! The old watchdog inferred the tail's state from a bare heartbeat timestamp,
4+
//! so "silent" had two indistinguishable causes: wedged in a syscall, or
5+
//! returned for good. A tail that gave up (journal permanently lost, rescan
6+
//! budget exhausted) then looked exactly like the stuck-IRP case the watchdog
7+
//! exists for, and it error-logged and cancelled I/O on the dead thread every
8+
//! scan, forever. The pulse makes the tail say which it is, and makes the
9+
//! "returned for good" report impossible to forget: it is published by a
10+
//! [`Drop`] guard, so every exit path, including a panic unwind and any
11+
//! `return` added later, retires the pulse.
12+
//!
13+
//! State and timestamp share one `u64` (tag in the top two bits, unix seconds
14+
//! below) so a reader always sees a consistent pair; two separate atomics
15+
//! could tear between a supervisor's loads.
16+
17+
use std::sync::atomic::{AtomicU64, Ordering};
18+
19+
const TAG_SHIFT: u32 = 62;
20+
const SECS_MASK: u64 = (1 << TAG_SHIFT) - 1;
21+
const TAG_BEAT: u64 = 0;
22+
const TAG_BUSY: u64 = 1;
23+
const TAG_RETIRED: u64 = 2;
24+
25+
/// One decoded pulse observation.
26+
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
27+
pub(crate) enum Pulse {
28+
/// The tail loop is turning; unix seconds of its last stamp.
29+
Beat(u64),
30+
/// The tail is inside a legitimately long, non-stamping operation (a full
31+
/// MFT rebuild); unix seconds of when it entered.
32+
Busy(u64),
33+
/// The tail thread returned. Not stuck, and not coming back on its own.
34+
Retired,
35+
}
36+
37+
/// The shared word a tail writes and its supervisor reads.
38+
///
39+
/// Ordering invariant: `Retired` must be the last value ever stored, and that
40+
/// is guaranteed structurally rather than atomically. Busy scopes live inside
41+
/// `tail_loop` while the retire guard wraps the whole call, so guard drop
42+
/// order publishes any pending `Beat` restore before `Retired`. This keeps the
43+
/// hot-path stamp a plain relaxed store instead of a compare-exchange.
44+
pub(crate) struct TailPulse(AtomicU64);
45+
46+
impl TailPulse {
47+
/// A fresh pulse, beating as of now (the tail is about to start).
48+
pub(crate) fn new() -> Self {
49+
Self(AtomicU64::new(pack(TAG_BEAT, now_unix())))
50+
}
51+
52+
/// Stamps liveness. Called once per tail-loop turn.
53+
pub(crate) fn beat(&self) {
54+
self.0.store(pack(TAG_BEAT, now_unix()), Ordering::Relaxed);
55+
}
56+
57+
/// The current observation.
58+
pub(crate) fn load(&self) -> Pulse {
59+
let word = self.0.load(Ordering::Relaxed);
60+
let secs = word & SECS_MASK;
61+
match word >> TAG_SHIFT {
62+
TAG_BUSY => Pulse::Busy(secs),
63+
TAG_RETIRED => Pulse::Retired,
64+
_ => Pulse::Beat(secs),
65+
}
66+
}
67+
68+
/// Marks the tail busy until the returned guard drops, which restores a
69+
/// fresh beat so a long rebuild does not read as an instant stall the
70+
/// moment it finishes. Scopes must not nest (the inner drop would end the
71+
/// outer scope early); the tail has exactly one, around `rescan`.
72+
pub(crate) fn busy_scope(&self) -> BusyGuard<'_> {
73+
self.0.store(pack(TAG_BUSY, now_unix()), Ordering::Relaxed);
74+
BusyGuard(self)
75+
}
76+
77+
/// Publishes `Retired` when the returned guard drops. Held across the
78+
/// whole `tail_loop` call so every exit, panic unwinds included, reports
79+
/// the thread as gone rather than leaving a heartbeat that merely went
80+
/// quiet.
81+
pub(crate) fn retire_on_drop(&self) -> RetireGuard<'_> {
82+
RetireGuard(self)
83+
}
84+
}
85+
86+
/// Restores a fresh [`Pulse::Beat`] on drop. See [`TailPulse::busy_scope`].
87+
pub(crate) struct BusyGuard<'a>(&'a TailPulse);
88+
89+
impl Drop for BusyGuard<'_> {
90+
fn drop(&mut self) {
91+
self.0.beat();
92+
}
93+
}
94+
95+
/// Publishes [`Pulse::Retired`] on drop. See [`TailPulse::retire_on_drop`].
96+
pub(crate) struct RetireGuard<'a>(&'a TailPulse);
97+
98+
impl Drop for RetireGuard<'_> {
99+
fn drop(&mut self) {
100+
self.0.0.store(pack(TAG_RETIRED, 0), Ordering::Relaxed);
101+
}
102+
}
103+
104+
fn pack(tag: u64, secs: u64) -> u64 {
105+
(tag << TAG_SHIFT) | (secs & SECS_MASK)
106+
}
107+
108+
pub(crate) fn now_unix() -> u64 {
109+
std::time::SystemTime::now()
110+
.duration_since(std::time::UNIX_EPOCH)
111+
.map(|d| d.as_secs())
112+
.unwrap_or(0)
113+
}
114+
115+
#[cfg(test)]
116+
mod tests {
117+
use super::*;
118+
119+
#[test]
120+
fn new_pulse_beats_now() {
121+
let p = TailPulse::new();
122+
let before = now_unix();
123+
match p.load() {
124+
Pulse::Beat(at) => assert!(at <= before && before - at <= 1),
125+
other => panic!("fresh pulse must beat, got {other:?}"),
126+
}
127+
}
128+
129+
#[test]
130+
fn busy_scope_restores_a_fresh_beat() {
131+
let p = TailPulse::new();
132+
{
133+
let _busy = p.busy_scope();
134+
assert!(matches!(p.load(), Pulse::Busy(_)));
135+
}
136+
assert!(matches!(p.load(), Pulse::Beat(_)));
137+
}
138+
139+
/// The design pillar: retirement is published by unwinding, not by
140+
/// remembering to call something, so a panicking tail still reports
141+
/// itself gone instead of leaving a heartbeat that merely went quiet.
142+
#[test]
143+
fn a_panicking_thread_still_retires() {
144+
let p = std::sync::Arc::new(TailPulse::new());
145+
let thread_p = p.clone();
146+
let joined = std::thread::spawn(move || {
147+
let _retired = thread_p.retire_on_drop();
148+
panic!("tail died unexpectedly");
149+
})
150+
.join();
151+
assert!(joined.is_err(), "the thread must have panicked");
152+
assert_eq!(p.load(), Pulse::Retired);
153+
}
154+
155+
/// Guard drop order is the whole ordering story: a busy scope inside the
156+
/// retire guard must end first, so `Retired` is the final word even
157+
/// though nothing enforces it atomically.
158+
#[test]
159+
fn retire_wins_over_an_unwinding_busy_scope() {
160+
let p = TailPulse::new();
161+
{
162+
let _retired = p.retire_on_drop();
163+
let _busy = p.busy_scope();
164+
// _busy drops first (restores Beat), then _retired publishes.
165+
}
166+
assert_eq!(p.load(), Pulse::Retired);
167+
}
168+
169+
/// Timestamps survive the tag bits for any plausible clock value,
170+
/// including far-future ones.
171+
#[test]
172+
fn tag_bits_never_corrupt_the_seconds() {
173+
for secs in [0u64, 1, 1_752_800_000, SECS_MASK] {
174+
assert_eq!(pack(TAG_BEAT, secs) & SECS_MASK, secs);
175+
assert_eq!(pack(TAG_BUSY, secs) >> TAG_SHIFT, TAG_BUSY);
176+
}
177+
}
178+
}

crates/goz-daemon/src/run.rs

Lines changed: 16 additions & 31 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@ use goz_core::types::VolumePhase;
1010
use parking_lot::RwLock;
1111
use tokio::sync::watch;
1212

13+
use crate::supervisor::Supervisor;
1314
use crate::tail::{self, TailHandle};
1415
use crate::volume_state::{VolumeSet, VolumeState};
1516
use crate::{bootstrap, server};
@@ -18,7 +19,8 @@ use crate::{bootstrap, server};
1819
/// exists) tailing live. Shared by console and service entry points.
1920
pub(crate) struct Engine {
2021
pub volumes: VolumeSet,
21-
pub tails: Vec<TailHandle>,
22+
/// Owns every tail thread; see [`Supervisor::shutdown`] for teardown.
23+
pub supervisor: Supervisor,
2224
pub total_entries: usize,
2325
pub elapsed: Duration,
2426
}
@@ -78,12 +80,14 @@ pub(crate) fn build_engine() -> Result<Engine> {
7880
entries = state.index.read().len(),
7981
"tailing journal"
8082
);
81-
match tail::spawn(state.clone(), bv.handle, cursor) {
83+
match tail::spawn(state.clone(), bv.handle, Some(cursor)) {
8284
Ok(h) => tails.push(h),
8385
Err(e) => {
84-
// OS thread creation failed (resource pressure). Degrade
85-
// just this volume to Offline instead of aborting the
86-
// whole engine bootstrap; status will flag it incomplete.
86+
// Thread creation or its handle duplication failed
87+
// (resource pressure); either way no tail is running.
88+
// Degrade just this volume to Offline instead of
89+
// aborting the whole engine bootstrap; status will
90+
// flag it incomplete.
8791
tracing::error!(
8892
volume = %state.mount_prefix(),
8993
error = %e,
@@ -110,28 +114,11 @@ pub(crate) fn build_engine() -> Result<Engine> {
110114
states.push(state);
111115
}
112116

113-
// One watchdog for every tail: the only defense against a device that
114-
// pends a synchronous journal IRP forever (observed on a spinning-down
115-
// HDD), which no amount of in-loop error handling can see.
116-
if !tails.is_empty() {
117-
let watched: Vec<_> = tails
118-
.iter()
119-
.map(|t| tail::WatchedTail {
120-
mount: t.mount.clone(),
121-
heartbeat: t.heartbeat.clone(),
122-
state: t.state.clone(),
123-
io: t.io_handle.clone(),
124-
})
125-
.collect();
126-
if let Err(e) = std::thread::Builder::new()
127-
.name("goz-tail-watchdog".into())
128-
.spawn(move || tail::watchdog_loop(watched))
129-
{
130-
// Degraded but not fatal: tails still run, they just lose the
131-
// stuck-I/O rescue.
132-
tracing::error!(error = %e, "failed to spawn tail watchdog");
133-
}
134-
}
117+
// One supervisor for every tail: cancels journal I/O that a sick device
118+
// pends forever (observed on a spinning-down HDD), which no in-loop error
119+
// handling can see, and revives tails that exit outright, on a bounded
120+
// budget. It takes ownership of the tails; teardown goes through it.
121+
let supervisor = Supervisor::spawn(tails);
135122

136123
// Purge THIS thread's allocator heap before handing off. The bootstrap
137124
// transients (sparse FRN maps swapped for dense, enumeration buffers,
@@ -145,7 +132,7 @@ pub(crate) fn build_engine() -> Result<Engine> {
145132

146133
Ok(Engine {
147134
volumes: Arc::new(states),
148-
tails,
135+
supervisor,
149136
total_entries,
150137
elapsed: started.elapsed(),
151138
})
@@ -216,8 +203,6 @@ pub(crate) fn run_console() -> Result<()> {
216203
let (_stop_tx, stop_rx) = watch::channel(false);
217204
let result = server::run(engine.volumes, stop_rx);
218205

219-
for t in engine.tails {
220-
t.shutdown();
221-
}
206+
engine.supervisor.shutdown();
222207
result
223208
}

crates/goz-daemon/src/service.rs

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -297,9 +297,7 @@ fn run_service() -> Result<()> {
297297
if let Err(e) = server::run(engine.volumes, stop_rx) {
298298
tracing::error!(error = %e, "pipe server error");
299299
}
300-
for t in engine.tails {
301-
t.shutdown();
302-
}
300+
engine.supervisor.shutdown();
303301
});
304302

305303
// Heartbeat StartPending until the engine reports ready (or fails).

0 commit comments

Comments
 (0)