TreeMap's native core lives here: a Cargo workspace, the hand-written declaration of what it exports to Node, and (once built) the module itself. It is an accelerator. The app never needs it to run: when the module cannot be loaded the legacy engines take over and the API says why.
native/
treemap-core/ the Cargo workspace (edition 2024, Rust ≥ 1.85)
crates/tm-governor the resource governor: presets, the closed loop, the OS mechanisms — pure Rust
crates/tm-walk the native walker: one listing call per directory, a governed walk, columnar output
crates/tm-store the finalized scan store, built in Rust from the walk's columns in PackedScanStore's own layout
crates/tm-mft the NTFS master file table, read as records (the NTFS turbo mode's reader; portable, tested on synthetic volumes)
crates/tm-mft-helper the turbo mode's elevated, read-only Windows helper: one volume's $MFT for one scan root, written as a columns file
crates/tm-node the napi-rs bindings — the only crate Node touches
index.d.ts the module's TypeScript declaration, the one source of truth for its shapes
THIRD_PARTY_NOTICES.txt the licences of every crate the binaries are built from (generated; see below)
licence-texts/<crate>/ upstream licence texts for a crate whose package carries none (kept by hand; none yet)
prebuilt/<platform>-<arch>/treemap_core.node the built module (gitignored)
prebuilt/<platform>-<arch>/VERSION the handshake value it was built for (gitignored)
You need Rust (https://rustup.rs). Then:
npm run build:native cargo build --release -p tm-node, copy the library to native/prebuilt/<platform>-<arch>/treemap_core.node, write VERSION beside it, print the path
npm run test:native cargo test --workspace (the Rust suites; the one-minute gate holds are #[ignore]d)
scripts/build-native.js runs cargo in native/treemap-core with cargo's own
target directory (native/treemap-core/target, or CARGO_TARGET_DIR when set)
and copies libtm_node.dylib / libtm_node.so / tm_node.dll from
<target>/release/. Without cargo it exits 1 with the install hint. Both
native/prebuilt/ and the target directory are gitignored: nothing built is
ever committed.
Users never build anything. CI (.github/workflows/test.yml) builds the
module on every leg — macOS, Windows, Linux — after cargo fmt --check,
cargo clippy --workspace --all-targets -- -D warnings and cargo test, and
the release workflow builds it before electron-builder. package.json's
build.files and build.asarUnpack both carry native/prebuilt/**, so the
bundle ships the module and Electron's process.dlopen redirect finds it in
app.asar.unpacked rather than inside the archive, which a dynamic loader
cannot open.
For iterating, a debug build is enough: cargo build -p tm-node, copy
target/debug/libtm_node.dylib (or .so / .dll) to any file named
treemap_core.node, and set TREEMAP_NATIVE_MODULE to that path. The loader
tries that path first, and tests/nativeLoader.test.ts uses it instead of
building. The name matters: Node only dlopens a .node file.
The Phase 2 gate — the governor holding 25, 50 and 90 percent of this machine
for a minute each — is
cargo test --release -p tm-governor --test hold -- --ignored --test-threads=1 --nocapture
in the Rust suite and npm run bench -- governor --preset=eco|balanced|turbo --seconds=60 --record
through the module (see bench/README.md).
treemap_core.node and tm-mft-helper.exe are statically linked from the
crates in treemap-core/Cargo.lock and from Rust's standard library, and MIT
and Apache-2.0 both ask for their notices to travel with every binary copy.
THIRD_PARTY_NOTICES.txt here holds them:
- every crate in Cargo.lock, with its licence expression, repository and the
licence files its own package carries, verbatim. Crates that only run while
the binaries are built (procedural macros, build-script dependencies) are
included too and marked
In the binaries: no. A crate whose package holds no licence file gets the text a person copied from its upstream repository intolicence-texts/<crate>/, or else the standard text of its licence, and its entry says which; - the standard library's own licences, and the notices of the crates.io crates
it is built from, as the pinned toolchain records them
(
share/doc/rust/COPYRIGHT-library.htmlandCOPYRIGHT.html). Which crates those are, per Rust version, isSTD_CRATESinscripts/rust-notices/toolchain.js; the generator checks the list against the/rust/deps/<crate>-<version>/paths in the installed standard libraries' rlibs.
package.json's build.extraResources ships it in the app's resources as
THIRD_PARTY_NOTICES.txt, beside gdu/LICENSE.md, which is what the About
panel points at.
It is generated, never edited by hand. After any change to Cargo.lock, to
rust-toolchain.toml (which also needs a STD_CRATES list for the new
version) or to the generator itself:
node scripts/rust-notices.js rewrite native/THIRD_PARTY_NOTICES.txt from cargo metadata and the toolchain (offline)
node scripts/rust-notices.js --check exit 1 if the committed file is not what it would write
It reads cargo metadata --offline --locked, so the crates must be in the
local registry (cargo fetch here puts them there, the one step that uses the
network). It refuses, writing nothing, when a licence cannot be met on the
permissive terms in PERMISSIVE (scripts/rust-notices/spdx.js): a copyleft
or unknown licence is a person's decision. The file's last line seals it with
the digest of the generator and of its own text. tests/rustNotices.test.ts
compares it with Cargo.lock, rust-toolchain.toml and the generator's source
alone, so it runs offline on every CI leg. It fails, naming what moved, when a
crate is added, removed or re-versioned without regenerating, when the
generator changed since the file was written, and when the file was edited by
hand. Bumping the workspace version needs no regeneration: the file names
TreeMap's own crates without versions.
gdu's notices (build/gdu/THIRD_PARTY_NOTICES.txt and
build/gdu/gdu-modules.json, written by scripts/gdu-notices.js and sealed
the same way) share two of these generator files: scripts/rust-notices/spdx.js
and scripts/rust-notices/text.js are named in both seals. A change to either
turns both tests/rustNotices.test.ts and tests/gduNotices.test.ts red, so
regenerate both:
node scripts/fetchGdu.js --all stage every target's gdu binary under build/gdu/ (network)
node scripts/gdu-notices.js rewrite gdu's notices from those binaries and the Go modules' source (network: proxy.golang.org)
node scripts/rust-notices.js rewrite native/THIRD_PARTY_NOTICES.txt (offline)
and commit the three files. scripts/fetchGdu.js never fetches
build/gdu/LICENSE.md again once it is there, so on a gdu version bump delete
it first; scripts/gdu-notices.js refuses one that is not the pinned
version's.
Three numbers must agree: nativeVersion in package.json, the workspace
version in native/treemap-core/Cargo.toml, and what the module's version()
returns (the crate version, so the second and third cannot differ).
scripts/build-native.jsrefuses to build whenpackage.jsonandCargo.tomldisagree, naming both.src/services/scan/native.tsrefuses to load a module whoseversion()is notnativeVersion, naming both, and the app runs without it.
Bump them together whenever the module's surface changes — that is, whenever
native/index.d.ts changes shape — so a prebuilt from an older contract can
never answer for a newer one.
loadNative() (src/services/scan/native.ts) tries, in order:
TREEMAP_NATIVE_MODULE, when set — and then nothing else: the variable is exclusive, so pointing it at a path that is not there forces the legacy engines (the reason names that path) and never falls through to a prebuilt;native/prebuilt/<platform>-<arch>/treemap_core.nodeunder the app root;<process.resourcesPath>/native/treemap_core.node(an Electron layout).
It never throws. The outcome is decided once per process and cached: either
{ available: true, module, version, path } or { available: false, reason }
with every candidate's reason joined — the file is not there for this
platform, dlopen refused it, it is not TreeMap's module, or its version is
not the one expected. Without the module src/services/engineBudget.ts
reports source: 'node-shim', holds the budget with fixed duties and worker
caps as far as Node allows, and GET /api/engine/capabilities lists every
mechanism as unavailable with that reason. Nothing is guessed: a number the
core did not measure is not shown.
The declaration is native/index.d.ts; src/services/engineBudget.ts
imports its types rather than re-declaring them. Every shape is the Rust type
serialised through serde with rename_all = "camelCase": enums as lower-case
strings, Option as null, shares as fractions of the whole machine (0..1).
| Export | What it does |
|---|---|
version() |
the crate version, for the handshake |
governorCapabilities() |
the seven mechanisms (qos, ioPolicy, priority, thermal, battery, interaction, machineCpu), each { available, mechanism, reason }; probed, never applied |
governorConfigure(budget, auto) |
sets the budget live: { preset: 'eco' | 'balanced' | 'turbo', cpuPercent?: 1..100 | null, autoBase?: 'eco' | 'balanced' | 'turbo' }; auto means the auto base — Balanced when absent, and Turbo while a scan a person started runs, which the app sends (contract 0.6.0) — one step lower on battery and Eco under serious heat; a bad budget throws and the one in force stays |
governorSnapshot() |
the Rust Snapshot: budget, effective preset, targetShare, share1s, workers, duty, thermal, battery, interaction, machineBusyShare, paused, ticks, the mechanisms last applied |
governorPause() / governorResume() |
block and release the workers; a thermal pause is the governor's own and stays |
governorHold(targetPercent, seconds) |
test-only: holds a percent of the machine with a synthetic load and resolves with the Rust HoldReport in shares, machineIdleLastHalf included (the whole machine's idle share over the second half, null where the OS published none: a hold under its target on a machine with no idle CPU measured the machine, not the governor) |
scanProbe(root) |
{ fastPath, reason }: opens and lists root once with this platform's listing (bulk on macOS; unavailable with the reason on a file, a root that cannot be read, or a platform whose listing is not built yet); never throws |
scanStart(root, opts) |
starts a walk on tm-walk's own threads, governed by the same process-wide governor, and returns a handle; opts is { neverDescend: string[], wantAtime: boolean, maxWorkers?: number, bufferBytes?: number, synthetic?: SyntheticSource } (0 or absent lets the hill-climber decide); refuses a root that is not a folder or cannot be read with Node's errno spelling in front (ENOENT: …), and a platform without a listing; with synthetic it lists a scripted tree instead of the disk (see "The synthetic source" below) |
syntheticTempFolder() |
the app's synthetic temp folder as the walk names it (TreeMap-synthetic in Rust's std::env::temp_dir()), inside which a synthetic root must lie; nothing is created |
scanPoll(handle) |
{ done, error, ceiling, entries, dirs, files, bytes, heartbeat, currentPath } from the walk's atomics — no callback, no ThreadsafeFunction (decision P3-1); heartbeat counts the batches the OS has answered, so it moves through one huge listing while entries cannot; once done, how the walk ended is in error, and ceiling is true when it ended at a ceiling its caller set — the memory store's room (contract 0.5.0) |
scanPause(handle) / scanResume(handle) |
stop the workers at their next check (between directories and every 256 entries inside one) and let them go on; nothing is re-listed |
scanCancel(handle) |
ends the walk; scanTake then throws the cancellation and frees the handle |
scanTake(handle) |
the columns (WalkResult in native/index.d.ts): typed arrays created from the Rust vectors without copying, the hard-link and refusal side tables, and the stats; frees the handle; throws in plain English when the walk failed or was cancelled (freeing it too) or when the handle is unknown; a memory-mode scan it frees and throws for, since its rows are in its store |
storeTake(handle) |
a memory-mode scan's store (NativeStore), as a promise, off the calling thread: PackedScanStore.adoptColumns' columns at capacity length — the store's own mappings in plain Node, copied where Electron refuses that — the candidate lists and sparse terms in breadth-first order, the counters and the stats; frees the handle; rejects with the walk's sentence when it failed or was cancelled, and refuses (keeping the handle) a walk still running, a scan without storage: 'memory' and an unknown handle (see "The memory mode" below) |
externalBuffersAllowed() |
whether this runtime lets a typed array be memory the addon owns (an N-API external buffer): plain Node does; Electron's memory cage does not, and there storeTake's arrays would be copied on the JavaScript thread (contract 0.4.0) |
storeShape(handle) |
the lengths of the arrays storeTakeInto fills (NativeStoreShape), once a poll has reported done; refused as storeTake refuses |
storeTakeInto(handle, arrays) |
a memory-mode scan's store copied into arrays JavaScript made as long as storeShape says, on libuv's pool, each column dropped right after its copy; frees the handle and resolves the NativeStore storeTake would, holding those arrays, with handOver: { bytesOffThread, bytesOnJsThread }; refuses, keeping the handle, as storeTake does and when an array is not the shape's length (Phase 4 T9c; takeNativeStore in nativeMemory.ts chooses it where external buffers are refused) |
The governor is one per process, started lazily on the first call with this
machine's sampler and signals, at the Automatic budget (the app's default);
the app configures it on every settings load, and moves its auto base whenever
a scan a person started begins or ends (contract 0.6.0). A walk obeys that same
governor: every worker calls its throttle after each directory and re-reads its
worker limit between directories, so the Scanning budget bounds the native
engine exactly as it bounds the legacy ones, and a budget that rises or falls
mid-walk reaches each worker at its next throttle. On Linux one part of a rise
does not: an unprivileged thread can raise its nice value but never lower it,
so a worker that already ran at a lower budget keeps that budget's priority
(the snapshot's mechanisms.priority says so) while its duty and the worker
count follow the new budget.
src/services/scan/nativeEngine.ts drives a scan: scanStart, then
scanPoll from 1 ms doubling up to 100 ms (so a scan of a few dozen entries
settles in a couple of milliseconds and a long one is polled at the SSE
cadence), scanPause/scanResume the instant the Phase 2 pause gate closes
or opens (the scan's Pausable), scanCancel on the record's cancel flag,
then scanTake and an ingest into the same PackedScanStore every engine
writes, through the same statToInput (src/services/scan/nodeInput.ts).
The columns are in discovery order with every parent before its children;
the ingest emits each directory's children byte-sorted by name off Windows,
because libuv's scandir sorts the legacy walker's listing that way while
getattrlistbulk returns APFS's own hash order — with that, the JSON the app
emits is byte-identical to the walker's on the same tree
(tests/nativeEngine.test.ts). cpuSeconds is the walk's own thread CPU
(CLOCK_THREAD_CPUTIME_ID) plus the ingest's process.cpuUsage() delta, or
null where the platform has no thread clock — never zero.
A handle that is never taken is freed when the process exits; scanTake on
an unknown handle throws.
A hold replaces only the ceiling — the configured preset, and with it the QoS
class and I/O policy the OS gives the workers, stays what it was — and hands
the budget back when it ends, unless the app reconfigured meanwhile (then the
newer budget stays). A second hold while one runs is refused, as is a target
outside 1..100 percent or a length outside 1..600 seconds, by rejection with a
message. The report's target is the share the governor was holding when the
hold began: the target asked for, scaled by 0.7 while the user is interacting
(Eco and Balanced) and by 0.5 under serious heat. The bench refuses a report
whose target is not the one it asked for, so run the gate on a quiet machine
you are not touching.
scanStart(root, { …, synthetic: { entries, … } }) walks a scripted tree
instead of the disk: tm-walk's SyntheticLister
(crates/tm-walk/src/platform/synthetic.rs) answers each listing from the
spec and the folder's path alone, so it creates, opens and reads nothing, and
one seed lists one tree at any worker count. It exists to measure the walk
and the store at sizes no disk here holds (the 100M-entry gate).
- Where.
rootmust lie strictly inside<temp>/TreeMap-synthetic/, by plain names (no..), where<temp>is Rust'sstd::env::temp_dir()(TMPDIR; on Windows TMP, then TEMP, then USERPROFILE) or its resolved path. Anything else is refused before a thread starts. Nothing is created there. Node askssyntheticTempFolder()for the folder:os.tmpdir()does not follow Rust's rule (with TMPDIR unset it reads TMP and TEMP, where Rust takes the per-user temp folder on macOS and/tmpon Linux). - The shape.
entriesunder the root (the root is not one, as the walk counts);folderShareof them, rounded down, are folders, fillingdepthlevels with at mostfanOutsubfolders each; the files are spread evenly over every folder. Every name isnameLengthbytes. Sizes are log-normal aroundsizeMedianwithsizeSigma, drawn with integer sums and IEEE 754's exactly rounded+ − × ÷ √only, so no platform's maths library enters a draw; allocated bytes are the size rounded up to 4,096.linkShareof the files, in whole pairs, are hard-linked. A pair's two names are half the files apart in file order, so they are in different folders unless one folder holds more than half the files: never with two or more subfolders; with none, every pair shares the root, and with one subfolder and an odd file count, one pair does. Every field butentriesdefaults to the developer shape of the Phase 4 design (§S.8): 16, 12, 0.15, 18, 4096, 2 and 0.01. - Refusals. A tree whose folders do not fit the fan-out and depth, names
too short to number the entries, a share outside 0..1, a sigma outside
0..10, more than 4,294,967,294 entries: each throws a sentence
(
WalkError::OptionsRefused), never a smaller tree. - Stats. Such a walk reports
fastPath: 'unavailable', since no platform listing ran. The bench labels its resultssource: syntheticandbench comparerefuses to set one beside a file-system run (bench/lib/report.ts, rule 8); the presets aresynthetic10mandsynthetic100minbench/lib/corpus.ts.
scanStart(root, { …, storage: 'memory', store: { … } }) (every scan the app
makes since Phase 4 T10) numbers the walk's entries in blocks, keeps no columns of
the walk's own, and feeds tm-store's MemorySink, which writes the finalized
store's columns while the walk runs and seals them on the walk's own thread
as it finishes (ListingSink::finish: the seal sees a cancel and moves the
heartbeat). store is tm-store's BuildOptions — rootName,
rootMtimeMs, blocksAreMeaningful, sortChildren, containerRules,
headroomRows — and the sink's room, capRows and nameBytes: a walk with
more entries, or more bytes of names, ends at its ceiling — the ceiling's
sentence in error and ceiling: true from scanPoll (contract 0.5.0) — and
the app walks that tree again on the columns path until Phase 4's T12 carries
such a walk on. storeTake hands the store
over; the sink travels with the scan's slot, so a scan freed any way releases
its store. The store's ids are the walk's blocks' (parent[id] < id, each
folder's children one range), not breadth-first.
governorHold is a napi AsyncTask, so its compute runs on libuv's thread
pool (four threads by default). That is acceptable for a test-only measurement
of a few seconds — one pool thread is busy for the hold's length, and Node's
own file work shares the rest — and it is not how a scan runs: a walk runs on
tm-walk's own threads (tm-walk-driver and tm-walk-worker-N, up to the
governor's limit) and Node reads its atomics through scanPoll, so libuv's
pool and the event loop are never in the walk's path. The governor's tick
thread is tm-governor-tick, the synthetic workers are tm-governor-load-N.
tm-node's unsafe is the store's hand-over in store.rs, in three places,
each with its SAFETY note: an anonymous column lent to a typed array
(with_external_data; the mapping moves into the array's finalizer), the
one-byte probe of whether this runtime takes external buffers
(napi_create_external_arraybuffer; the byte is freed by the finalizer, or at
once where the buffer is refused), and storeTakeInto's borrows of the arrays
JavaScript passed in (as_mut; nothing else touches them until its promise
settles). On Windows one more: pin.rs pins the DLL as it is loaded
(GetModuleHandleExW with GET_MODULE_HANDLE_EX_FLAG_PIN), because Node
closes every addon an environment loaded when the environment is destroyed,
and where a worker thread was the addon's only holder FreeLibrary unmapped it
while the worker's teardown still called into it: the process died with an
access violation (RISKS R96). Pinned, the DLL stays mapped until the process
ends, whichever thread loaded it first. Every export is #[napi(catch_unwind)], and
the hold's compute wraps the governor the same way, because libuv calls it
through an extern "C" boundary that a panic must never cross; a caught panic
becomes a JavaScript error with its message. The workspace lints deny
unwrap, expect, panic, indexing_slicing, todo, unimplemented and
unreachable, and clippy::pedantic is on. Nothing in this workspace touches
a GPU.
Windows and Linux are compile-checked here with
cargo check --target x86_64-pc-windows-msvc and
cargo check --target x86_64-unknown-linux-gnu and built for real by CI.
napi-build adds the linker arguments a Node addon needs on each platform;
napi-rs resolves the N-API symbols from the host at load, so no node.lib is
needed on Windows.