Upstream tracking - #165
Draft
grahamc wants to merge 4192 commits into
Draft
Conversation
…d52-93b9-4419-9060-8a0e0eb8c8c1 Release v3.22.2
Previously, running out of memory resulted in an opaque SIGABRT (e.g. Sentry issue DETERMINATE-NIX-8P): mimalloc's `operator new` override is compiled as C, so it cannot throw `std::bad_alloc` and instead calls abort() when an allocation fails and no new handler is installed. So install a `std::new_handler` that panics with a "ran out of memory" message. However, that alone doesn't help under mimalloc: libmimalloc is linked with `-Bsymbolic`, so its weak null stub of `std::get_new_handler()` shadows the real one from libstdc++ and the installed handler is never seen. Therefore also register a mimalloc error callback via `mi_register_error()` that panics on ENOMEM/EOVERFLOW. This also sets the `panic_msg` Sentry tag, so OOM crash reports now self-identify instead of showing up as unexplained aborts in allocation paths. Assisted-by: Claude Fable 5 <noreply@anthropic.com>
We have no ability to meaningfully handle bad_alloc, and it can lead to terminate() calls when it happens in a destructor or some other context where throwing is not allowed. So just panic with a clear "ran out of memory" message. Assisted-by: Claude Fable 5 <noreply@anthropic.com>
Allocation failures now call outOfMemory(), which panics instead of throwing bad_alloc, so this catch is unreachable. Assisted-by: Claude Fable 5 <noreply@anthropic.com>
Errors that occur while interpreting a syntactically valid flake ref URL (such as an unsupported query parameter) were swallowed by parseURLFlakeRef(), causing the flake ref to be reinterpreted as a path and leading to a confusing error, e.g. $ nix --option nix-219-compat true eval --expr 'builtins.parseFlakeRef "github:NixOS/nix?dir=perl"' error: flake reference 'github:NixOS/nix?dir=perl' is not an absolute path instead of the underlying error error: URL 'github:NixOS/nix?dir=perl' contains unknown parameter 'dir' Now we only fall back to path parsing if the string doesn't parse as a URL at all. Assisted-by: Claude Fable 5 <noreply@anthropic.com>
Since f3aeeca, nix-219-compat mode kept the 'dir' query parameter in the URL passed to Input::fromURL(), in order to reproduce the 'url' attribute that Nix < 2.20 wrote to lock files (e.g. "url": "https://foo/bar?dir=subdir"). However, since 6cae299, the GitHub input scheme rejects unknown query parameters, so flake refs like 'github:NixOS/nix?dir=perl' failed to parse in compat mode. Nix 2.19.0 only retained 'dir' in the 'url' attribute of input types that have one (git, hg, tarball, file); for input types like 'github', 'dir' was never part of the input attributes. So instead of keeping 'dir' in the query (which input schemes may reject), we now always erase it before calling Input::fromURL(), and in compat mode re-add it to the resulting 'url' attribute if the input has one. Verified against Nix 2.19.0 that 'builtins.parseFlakeRef' now produces identical attributes in compat mode for git, hg, tarball, file, github, gitlab and sourcehut flake refs with a 'dir' parameter, and that lock files generated in compat mode are identical to those generated by 2.19.0 for a 'git+file://...?dir=sub' input. Assisted-by: Claude Fable 5 <noreply@anthropic.com>
Fix `dir` attribute handling in nix-219-compat mode
Write the schema version to a temporary file, fsync it, and atomically rename it into place. This ensures the schema file is never observed empty or partially written. Assisted-by: Claude Fable 5 <noreply@anthropic.com>
Update the schema file atomically
Profiling 'nix search nixpkgs --no-eval-cache' showed that ~65% of all futex waits came from the Boehm GC, the largest share being the global allocation lock (GC_allocate_ml): with the default GC_TINY_FREELISTS of 25, only allocations up to 384 bytes are served from the per-thread freelists, and everything larger takes the global lock. Nixpkgs evaluation does ~755k such allocations per search — e.g. a typical derivation attrset (~46 attrs) is a 752-byte Bindings, of which one is allocated per package — all serialized on the lock across the worker threads. Building bdwgc with GC_TINY_FREELISTS=96 (thread-local up to 1520 bytes) eliminates 93% of the global-lock allocations (755k -> 56k; what remains is essentially the >16 KiB attrset giants), halves the context-switch count at 8-16 workers, and makes the search ~10% faster at 24 workers (~3.7s -> ~3.3s). The per-thread memory cost is at most a partial heap block per size class and object kind. Also add a patch (from the bdwgc "nix-patches" branch) that makes gctest's fixed "unexpected heap growth" limit scale with GC_TINY_FREELISTS, since the larger freelists exceed it by design. Assisted-by: Claude Fable 5 <noreply@anthropic.com>
…ists Reduce contention on the bdwgc global lock
The JSON parser used GC-visible (uncollectable) containers in every object/array parser state (ValueMap/ValueVector) to keep the values under construction alive. Since GC_MALLOC_UNCOLLECTABLE() and GC_FREE() take the global GC allocation lock, a big JSON document did one or more global-lock round trips per object/array node. Profiling 'nix search nixpkgs --no-eval-cache' showed ~183k such allocations (nixpkgs evaluation parses a lot of JSON), making this the largest remaining source of allocation-lock contention after GC_TINY_FREELISTS was increased. Instead, root every value allocated during the parse in a single GC-visible vector for the duration of the parse (they all end up in the result anyway), and use ordinary (malloc) containers in the per-node parser states. This reduces the parser's uncollectable allocations from ~183k to zero, cutting the context-switch count of the search benchmark by another ~15% at 24 eval cores. Assisted-by: Claude Fable 5 <noreply@anthropic.com>
BindingsBuilder sorts the attributes anyway, so hashing every key on insertion is unnecessary. Keys are now appended to a vector; on object close, a stable sort plus a keep-last-of-run scan preserves JSON's last-occurrence-wins semantics for duplicate keys, and the pre-sorted result lets us use BindingsBuilder::alreadySorted() to skip the builder's own sort. Assisted-by: Claude Fable 5 <noreply@anthropic.com>
Small JSON objects and lists (up to 16 entries) now build up in the state's inline storage instead of a separately heap-allocated vector. Also drop the speculative reserve(128) for arrays: nlohmann's text parser never knows the length in advance (only binary formats like CBOR do), so it only served to force a heap allocation. Assisted-by: Claude Fable 5 <noreply@anthropic.com>
fromJSON: Eliminate use of uncollectable containers
This reverts commit deb3de4. This was a bad idea because it broke RemoteStore::addTempRoot() entirely.
nix copy: Don't register temp roots if the remote doesn't have AddTempRoots
Flake lock file updates:
• Updated input 'nixpkgs':
'https://api.flakehub.com/f/pinned/NixOS/nixpkgs/0.2605.1009383%2Brev-4382ed2b7a6839d4280a9b386db49cbc5907414d/019f6c11-ad78-7500-a194-d18a5bad0fbe/source.tar.gz' (2026-07-16)
→ 'https://api.flakehub.com/f/pinned/NixOS/nixpkgs/0.2605.1012930%2Brev-c5c4a43b0e8056328ec4529f735cabdb8f1942bb/01a053c0-7061-7382-b001-102e9148a5b5/source.tar.gz' (2026-08-29)
Flake lock file updates:
• Updated input 'nix/nixpkgs':
'https://api.flakehub.com/f/pinned/DeterminateSystems/secure-packages-26.05/0.1.1012125%2Brev-86c2aa5aa3e08175d183edb2581a7fd32c358115/01a01b96-88bb-7c70-84ca-1cbd5a037ecf/source.tar.gz' (2026-08-19)
→ 'https://api.flakehub.com/f/pinned/DeterminateSystems/secure-packages-26.05/0.1.1013487%2Brev-01649628bfcaf28024737daec63d559fe0463e6c/01a04eb4-9c90-76b3-a3fb-a92559e04618/source.tar.gz' (2026-08-29)
The GC roots server's per-client thread only caught Error, but Interrupted derives from BaseError, not Error. So when the daemon was interrupted during garbage collection while a client thread was blocked in readLine(), the Interrupted exception escaped the thread, calling std::terminate() and crashing the daemon. Fixes Sentry report DETERMINATE-NIX-31. Assisted-by: Claude Fable 5 <noreply@anthropic.com>
cache.nixos.org compresses newer NARs with zstd instead of xz, but the script unconditionally piped downloaded debuginfo NARs through `xz -d`. This broke after the Nixpkgs update pulled in glibc 2.42, whose debuginfo NAR is served as .nar.zst. Pick the decompressor based on the archive extension, and add xz/zstd to the nix shell shebang so both are guaranteed to be on PATH. Assisted-by: Claude Fable 5 <noreply@anthropic.com>
Update Nixpkgs
Fix crash when GC is interrupted
Move the mi_register_error() call before initLibUtil() so that allocation failures during initialization also invoke outOfMemory() instead of mimalloc's silent abort. Also handle EFAULT (heap corruption detected in secure mode) by calling panic(), preserving fatal behavior but with a proper error message. Assisted-by: Claude Fable 5 <noreply@anthropic.com>
Handle out-of-memory crashes better
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
10 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Not intended to be merged directly. This PR is a convenience to show the diff between upstream Nix and Determinate Nix (the
mainbranch).Continuation of #4.