Skip to content

Latest commit

 

History

History
331 lines (286 loc) · 22.9 KB

README.md

File metadata and controls

331 lines (286 loc) · 22.9 KB

native — the Rust core

TreeMap's native core lives here: a Cargo workspace, the hand-written declaration of what it exports to Node, and (once built) the module itself. It is an accelerator. The app never needs it to run: when the module cannot be loaded the legacy engines take over and the API says why.

native/
  treemap-core/          the Cargo workspace (edition 2024, Rust ≥ 1.85)
    crates/tm-governor   the resource governor: presets, the closed loop, the OS mechanisms — pure Rust
    crates/tm-walk       the native walker: one listing call per directory, a governed walk, columnar output
    crates/tm-store      the finalized scan store, built in Rust from the walk's columns in PackedScanStore's own layout
    crates/tm-mft        the NTFS master file table, read as records (the NTFS turbo mode's reader; portable, tested on synthetic volumes)
    crates/tm-mft-helper the turbo mode's elevated, read-only Windows helper: one volume's $MFT for one scan root, written as a columns file
    crates/tm-node       the napi-rs bindings — the only crate Node touches
  index.d.ts             the module's TypeScript declaration, the one source of truth for its shapes
  THIRD_PARTY_NOTICES.txt  the licences of every crate the binaries are built from (generated; see below)
  licence-texts/<crate>/   upstream licence texts for a crate whose package carries none (kept by hand; none yet)
  prebuilt/<platform>-<arch>/treemap_core.node   the built module (gitignored)
  prebuilt/<platform>-<arch>/VERSION             the handshake value it was built for (gitignored)

Building

You need Rust (https://rustup.rs). Then:

npm run build:native      cargo build --release -p tm-node, copy the library to native/prebuilt/<platform>-<arch>/treemap_core.node, write VERSION beside it, print the path
npm run test:native       cargo test --workspace (the Rust suites; the one-minute gate holds are #[ignore]d)

scripts/build-native.js runs cargo in native/treemap-core with cargo's own target directory (native/treemap-core/target, or CARGO_TARGET_DIR when set) and copies libtm_node.dylib / libtm_node.so / tm_node.dll from <target>/release/. Without cargo it exits 1 with the install hint. Both native/prebuilt/ and the target directory are gitignored: nothing built is ever committed.

Users never build anything. CI (.github/workflows/test.yml) builds the module on every leg — macOS, Windows, Linux — after cargo fmt --check, cargo clippy --workspace --all-targets -- -D warnings and cargo test, and the release workflow builds it before electron-builder. package.json's build.files and build.asarUnpack both carry native/prebuilt/**, so the bundle ships the module and Electron's process.dlopen redirect finds it in app.asar.unpacked rather than inside the archive, which a dynamic loader cannot open.

For iterating, a debug build is enough: cargo build -p tm-node, copy target/debug/libtm_node.dylib (or .so / .dll) to any file named treemap_core.node, and set TREEMAP_NATIVE_MODULE to that path. The loader tries that path first, and tests/nativeLoader.test.ts uses it instead of building. The name matters: Node only dlopens a .node file.

The Phase 2 gate — the governor holding 25, 50 and 90 percent of this machine for a minute each — is cargo test --release -p tm-governor --test hold -- --ignored --test-threads=1 --nocapture in the Rust suite and npm run bench -- governor --preset=eco|balanced|turbo --seconds=60 --record through the module (see bench/README.md).

Third-party notices

treemap_core.node and tm-mft-helper.exe are statically linked from the crates in treemap-core/Cargo.lock and from Rust's standard library, and MIT and Apache-2.0 both ask for their notices to travel with every binary copy. THIRD_PARTY_NOTICES.txt here holds them:

  • every crate in Cargo.lock, with its licence expression, repository and the licence files its own package carries, verbatim. Crates that only run while the binaries are built (procedural macros, build-script dependencies) are included too and marked In the binaries: no. A crate whose package holds no licence file gets the text a person copied from its upstream repository into licence-texts/<crate>/, or else the standard text of its licence, and its entry says which;
  • the standard library's own licences, and the notices of the crates.io crates it is built from, as the pinned toolchain records them (share/doc/rust/COPYRIGHT-library.html and COPYRIGHT.html). Which crates those are, per Rust version, is STD_CRATES in scripts/rust-notices/toolchain.js; the generator checks the list against the /rust/deps/<crate>-<version>/ paths in the installed standard libraries' rlibs.

package.json's build.extraResources ships it in the app's resources as THIRD_PARTY_NOTICES.txt, beside gdu/LICENSE.md, which is what the About panel points at.

It is generated, never edited by hand. After any change to Cargo.lock, to rust-toolchain.toml (which also needs a STD_CRATES list for the new version) or to the generator itself:

node scripts/rust-notices.js           rewrite native/THIRD_PARTY_NOTICES.txt from cargo metadata and the toolchain (offline)
node scripts/rust-notices.js --check   exit 1 if the committed file is not what it would write

It reads cargo metadata --offline --locked, so the crates must be in the local registry (cargo fetch here puts them there, the one step that uses the network). It refuses, writing nothing, when a licence cannot be met on the permissive terms in PERMISSIVE (scripts/rust-notices/spdx.js): a copyleft or unknown licence is a person's decision. The file's last line seals it with the digest of the generator and of its own text. tests/rustNotices.test.ts compares it with Cargo.lock, rust-toolchain.toml and the generator's source alone, so it runs offline on every CI leg. It fails, naming what moved, when a crate is added, removed or re-versioned without regenerating, when the generator changed since the file was written, and when the file was edited by hand. Bumping the workspace version needs no regeneration: the file names TreeMap's own crates without versions.

gdu's notices (build/gdu/THIRD_PARTY_NOTICES.txt and build/gdu/gdu-modules.json, written by scripts/gdu-notices.js and sealed the same way) share two of these generator files: scripts/rust-notices/spdx.js and scripts/rust-notices/text.js are named in both seals. A change to either turns both tests/rustNotices.test.ts and tests/gduNotices.test.ts red, so regenerate both:

node scripts/fetchGdu.js --all     stage every target's gdu binary under build/gdu/ (network)
node scripts/gdu-notices.js        rewrite gdu's notices from those binaries and the Go modules' source (network: proxy.golang.org)
node scripts/rust-notices.js       rewrite native/THIRD_PARTY_NOTICES.txt (offline)

and commit the three files. scripts/fetchGdu.js never fetches build/gdu/LICENSE.md again once it is there, so on a gdu version bump delete it first; scripts/gdu-notices.js refuses one that is not the pinned version's.

The version handshake

Three numbers must agree: nativeVersion in package.json, the workspace version in native/treemap-core/Cargo.toml, and what the module's version() returns (the crate version, so the second and third cannot differ).

  • scripts/build-native.js refuses to build when package.json and Cargo.toml disagree, naming both.
  • src/services/scan/native.ts refuses to load a module whose version() is not nativeVersion, naming both, and the app runs without it.

Bump them together whenever the module's surface changes — that is, whenever native/index.d.ts changes shape — so a prebuilt from an older contract can never answer for a newer one.

The fallback

loadNative() (src/services/scan/native.ts) tries, in order:

  1. TREEMAP_NATIVE_MODULE, when set — and then nothing else: the variable is exclusive, so pointing it at a path that is not there forces the legacy engines (the reason names that path) and never falls through to a prebuilt;
  2. native/prebuilt/<platform>-<arch>/treemap_core.node under the app root;
  3. <process.resourcesPath>/native/treemap_core.node (an Electron layout).

It never throws. The outcome is decided once per process and cached: either { available: true, module, version, path } or { available: false, reason } with every candidate's reason joined — the file is not there for this platform, dlopen refused it, it is not TreeMap's module, or its version is not the one expected. Without the module src/services/engineBudget.ts reports source: 'node-shim', holds the budget with fixed duties and worker caps as far as Node allows, and GET /api/engine/capabilities lists every mechanism as unavailable with that reason. Nothing is guessed: a number the core did not measure is not shown.

What the module exports

The declaration is native/index.d.ts; src/services/engineBudget.ts imports its types rather than re-declaring them. Every shape is the Rust type serialised through serde with rename_all = "camelCase": enums as lower-case strings, Option as null, shares as fractions of the whole machine (0..1).

Export What it does
version() the crate version, for the handshake
governorCapabilities() the seven mechanisms (qos, ioPolicy, priority, thermal, battery, interaction, machineCpu), each { available, mechanism, reason }; probed, never applied
governorConfigure(budget, auto) sets the budget live: { preset: 'eco' | 'balanced' | 'turbo', cpuPercent?: 1..100 | null, autoBase?: 'eco' | 'balanced' | 'turbo' }; auto means the auto base — Balanced when absent, and Turbo while a scan a person started runs, which the app sends (contract 0.6.0) — one step lower on battery and Eco under serious heat; a bad budget throws and the one in force stays
governorSnapshot() the Rust Snapshot: budget, effective preset, targetShare, share1s, workers, duty, thermal, battery, interaction, machineBusyShare, paused, ticks, the mechanisms last applied
governorPause() / governorResume() block and release the workers; a thermal pause is the governor's own and stays
governorHold(targetPercent, seconds) test-only: holds a percent of the machine with a synthetic load and resolves with the Rust HoldReport in shares, machineIdleLastHalf included (the whole machine's idle share over the second half, null where the OS published none: a hold under its target on a machine with no idle CPU measured the machine, not the governor)
scanProbe(root) { fastPath, reason }: opens and lists root once with this platform's listing (bulk on macOS; unavailable with the reason on a file, a root that cannot be read, or a platform whose listing is not built yet); never throws
scanStart(root, opts) starts a walk on tm-walk's own threads, governed by the same process-wide governor, and returns a handle; opts is { neverDescend: string[], wantAtime: boolean, maxWorkers?: number, bufferBytes?: number, synthetic?: SyntheticSource } (0 or absent lets the hill-climber decide); refuses a root that is not a folder or cannot be read with Node's errno spelling in front (ENOENT: …), and a platform without a listing; with synthetic it lists a scripted tree instead of the disk (see "The synthetic source" below)
syntheticTempFolder() the app's synthetic temp folder as the walk names it (TreeMap-synthetic in Rust's std::env::temp_dir()), inside which a synthetic root must lie; nothing is created
scanPoll(handle) { done, error, ceiling, entries, dirs, files, bytes, heartbeat, currentPath } from the walk's atomics — no callback, no ThreadsafeFunction (decision P3-1); heartbeat counts the batches the OS has answered, so it moves through one huge listing while entries cannot; once done, how the walk ended is in error, and ceiling is true when it ended at a ceiling its caller set — the memory store's room (contract 0.5.0)
scanPause(handle) / scanResume(handle) stop the workers at their next check (between directories and every 256 entries inside one) and let them go on; nothing is re-listed
scanCancel(handle) ends the walk; scanTake then throws the cancellation and frees the handle
scanTake(handle) the columns (WalkResult in native/index.d.ts): typed arrays created from the Rust vectors without copying, the hard-link and refusal side tables, and the stats; frees the handle; throws in plain English when the walk failed or was cancelled (freeing it too) or when the handle is unknown; a memory-mode scan it frees and throws for, since its rows are in its store
storeTake(handle) a memory-mode scan's store (NativeStore), as a promise, off the calling thread: PackedScanStore.adoptColumns' columns at capacity length — the store's own mappings in plain Node, copied where Electron refuses that — the candidate lists and sparse terms in breadth-first order, the counters and the stats; frees the handle; rejects with the walk's sentence when it failed or was cancelled, and refuses (keeping the handle) a walk still running, a scan without storage: 'memory' and an unknown handle (see "The memory mode" below)
externalBuffersAllowed() whether this runtime lets a typed array be memory the addon owns (an N-API external buffer): plain Node does; Electron's memory cage does not, and there storeTake's arrays would be copied on the JavaScript thread (contract 0.4.0)
storeShape(handle) the lengths of the arrays storeTakeInto fills (NativeStoreShape), once a poll has reported done; refused as storeTake refuses
storeTakeInto(handle, arrays) a memory-mode scan's store copied into arrays JavaScript made as long as storeShape says, on libuv's pool, each column dropped right after its copy; frees the handle and resolves the NativeStore storeTake would, holding those arrays, with handOver: { bytesOffThread, bytesOnJsThread }; refuses, keeping the handle, as storeTake does and when an array is not the shape's length (Phase 4 T9c; takeNativeStore in nativeMemory.ts chooses it where external buffers are refused)

The governor is one per process, started lazily on the first call with this machine's sampler and signals, at the Automatic budget (the app's default); the app configures it on every settings load, and moves its auto base whenever a scan a person started begins or ends (contract 0.6.0). A walk obeys that same governor: every worker calls its throttle after each directory and re-reads its worker limit between directories, so the Scanning budget bounds the native engine exactly as it bounds the legacy ones, and a budget that rises or falls mid-walk reaches each worker at its next throttle. On Linux one part of a rise does not: an unprivileged thread can raise its nice value but never lower it, so a worker that already ran at a lower budget keeps that budget's priority (the snapshot's mechanisms.priority says so) while its duty and the worker count follow the new budget.

The native walker (Phase 3)

src/services/scan/nativeEngine.ts drives a scan: scanStart, then scanPoll from 1 ms doubling up to 100 ms (so a scan of a few dozen entries settles in a couple of milliseconds and a long one is polled at the SSE cadence), scanPause/scanResume the instant the Phase 2 pause gate closes or opens (the scan's Pausable), scanCancel on the record's cancel flag, then scanTake and an ingest into the same PackedScanStore every engine writes, through the same statToInput (src/services/scan/nodeInput.ts). The columns are in discovery order with every parent before its children; the ingest emits each directory's children byte-sorted by name off Windows, because libuv's scandir sorts the legacy walker's listing that way while getattrlistbulk returns APFS's own hash order — with that, the JSON the app emits is byte-identical to the walker's on the same tree (tests/nativeEngine.test.ts). cpuSeconds is the walk's own thread CPU (CLOCK_THREAD_CPUTIME_ID) plus the ingest's process.cpuUsage() delta, or null where the platform has no thread clock — never zero.

A handle that is never taken is freed when the process exits; scanTake on an unknown handle throws.

A hold replaces only the ceiling — the configured preset, and with it the QoS class and I/O policy the OS gives the workers, stays what it was — and hands the budget back when it ends, unless the app reconfigured meanwhile (then the newer budget stays). A second hold while one runs is refused, as is a target outside 1..100 percent or a length outside 1..600 seconds, by rejection with a message. The report's target is the share the governor was holding when the hold began: the target asked for, scaled by 0.7 while the user is interacting (Eco and Balanced) and by 0.5 under serious heat. The bench refuses a report whose target is not the one it asked for, so run the gate on a quiet machine you are not touching.

The synthetic source (Phase 4, P4-8)

scanStart(root, { …, synthetic: { entries, … } }) walks a scripted tree instead of the disk: tm-walk's SyntheticLister (crates/tm-walk/src/platform/synthetic.rs) answers each listing from the spec and the folder's path alone, so it creates, opens and reads nothing, and one seed lists one tree at any worker count. It exists to measure the walk and the store at sizes no disk here holds (the 100M-entry gate).

  • Where. root must lie strictly inside <temp>/TreeMap-synthetic/, by plain names (no ..), where <temp> is Rust's std::env::temp_dir() (TMPDIR; on Windows TMP, then TEMP, then USERPROFILE) or its resolved path. Anything else is refused before a thread starts. Nothing is created there. Node asks syntheticTempFolder() for the folder: os.tmpdir() does not follow Rust's rule (with TMPDIR unset it reads TMP and TEMP, where Rust takes the per-user temp folder on macOS and /tmp on Linux).
  • The shape. entries under the root (the root is not one, as the walk counts); folderShare of them, rounded down, are folders, filling depth levels with at most fanOut subfolders each; the files are spread evenly over every folder. Every name is nameLength bytes. Sizes are log-normal around sizeMedian with sizeSigma, drawn with integer sums and IEEE 754's exactly rounded + − × ÷ √ only, so no platform's maths library enters a draw; allocated bytes are the size rounded up to 4,096. linkShare of the files, in whole pairs, are hard-linked. A pair's two names are half the files apart in file order, so they are in different folders unless one folder holds more than half the files: never with two or more subfolders; with none, every pair shares the root, and with one subfolder and an odd file count, one pair does. Every field but entries defaults to the developer shape of the Phase 4 design (§S.8): 16, 12, 0.15, 18, 4096, 2 and 0.01.
  • Refusals. A tree whose folders do not fit the fan-out and depth, names too short to number the entries, a share outside 0..1, a sigma outside 0..10, more than 4,294,967,294 entries: each throws a sentence (WalkError::OptionsRefused), never a smaller tree.
  • Stats. Such a walk reports fastPath: 'unavailable', since no platform listing ran. The bench labels its results source: synthetic and bench compare refuses to set one beside a file-system run (bench/lib/report.ts, rule 8); the presets are synthetic10m and synthetic100m in bench/lib/corpus.ts.

The memory mode (Phase 4, T8b)

scanStart(root, { …, storage: 'memory', store: { … } }) (every scan the app makes since Phase 4 T10) numbers the walk's entries in blocks, keeps no columns of the walk's own, and feeds tm-store's MemorySink, which writes the finalized store's columns while the walk runs and seals them on the walk's own thread as it finishes (ListingSink::finish: the seal sees a cancel and moves the heartbeat). store is tm-store's BuildOptions — rootName, rootMtimeMs, blocksAreMeaningful, sortChildren, containerRules, headroomRows — and the sink's room, capRows and nameBytes: a walk with more entries, or more bytes of names, ends at its ceiling — the ceiling's sentence in error and ceiling: true from scanPoll (contract 0.5.0) — and the app walks that tree again on the columns path until Phase 4's T12 carries such a walk on. storeTake hands the store over; the sink travels with the scan's slot, so a scan freed any way releases its store. The store's ids are the walk's blocks' (parent[id] < id, each folder's children one range), not breadth-first.

Threads

governorHold is a napi AsyncTask, so its compute runs on libuv's thread pool (four threads by default). That is acceptable for a test-only measurement of a few seconds — one pool thread is busy for the hold's length, and Node's own file work shares the rest — and it is not how a scan runs: a walk runs on tm-walk's own threads (tm-walk-driver and tm-walk-worker-N, up to the governor's limit) and Node reads its atomics through scanPoll, so libuv's pool and the event loop are never in the walk's path. The governor's tick thread is tm-governor-tick, the synthetic workers are tm-governor-load-N.

Safety and lints

tm-node's unsafe is the store's hand-over in store.rs, in three places, each with its SAFETY note: an anonymous column lent to a typed array (with_external_data; the mapping moves into the array's finalizer), the one-byte probe of whether this runtime takes external buffers (napi_create_external_arraybuffer; the byte is freed by the finalizer, or at once where the buffer is refused), and storeTakeInto's borrows of the arrays JavaScript passed in (as_mut; nothing else touches them until its promise settles). On Windows one more: pin.rs pins the DLL as it is loaded (GetModuleHandleExW with GET_MODULE_HANDLE_EX_FLAG_PIN), because Node closes every addon an environment loaded when the environment is destroyed, and where a worker thread was the addon's only holder FreeLibrary unmapped it while the worker's teardown still called into it: the process died with an access violation (RISKS R96). Pinned, the DLL stays mapped until the process ends, whichever thread loaded it first. Every export is #[napi(catch_unwind)], and the hold's compute wraps the governor the same way, because libuv calls it through an extern "C" boundary that a panic must never cross; a caught panic becomes a JavaScript error with its message. The workspace lints deny unwrap, expect, panic, indexing_slicing, todo, unimplemented and unreachable, and clippy::pedantic is on. Nothing in this workspace touches a GPU.

Windows and Linux are compile-checked here with cargo check --target x86_64-pc-windows-msvc and cargo check --target x86_64-unknown-linux-gnu and built for real by CI. napi-build adds the linker arguments a Node addon needs on each platform; napi-rs resolves the N-API symbols from the host at load, so no node.lib is needed on Windows.