fetchers: tarball cache used past ttl

[tomberek]

This is to fix a reported issue that fetchTarball with hashes were being re-fetched when TTLs expired. This adds another path to check if treeHash + narHash is still cached and uses it even if TTL is expired (except with TTL == 0 indicating a --refresh was used).

Another approach could be to explicitly add a narHash -> treeHash row. This seemed simpler for now.

Code was Co-Authored-By: Claude Sonnet 5 noreply@anthropic.com

[figsoda]

thanks, this does fix the exact issue i was running into

one small question, it seems like it still tries to substitute this path every eval, is that expected?

$ nix eval -f . --option tarball-ttl 1 -vvvv
evaluating file '«nix-internal»/derivation-internal.nix'
evaluating file '/home/figsoda/cfg/default.nix'
performing daemon worker op: 10
substitution of '/nix/store/96zbdkrn93l6qjwq23xpxifqm8qcld3k-source': created
substitution of '/nix/store/96zbdkrn93l6qjwq23xpxifqm8qcld3k-source': woken up
querying info about missing paths...
starting pool of 24 threads
checking substituter 'https://cache.nixos.org' for path '/nix/store/96zbdkrn93l6qjwq23xpxifqm8qcld3k-source'
checking substituter 'https://fenix.cachix.org' for path '/nix/store/96zbdkrn93l6qjwq23xpxifqm8qcld3k-source'
checking substituter 'https://nix-community.cachix.org' for path '/nix/store/96zbdkrn93l6qjwq23xpxifqm8qcld3k-source'
entered goal loop
substitution of '/nix/store/96zbdkrn93l6qjwq23xpxifqm8qcld3k-source': init
acquiring write lock on "/nix/var/nix/temproots/3256124"
substitution of '/nix/store/96zbdkrn93l6qjwq23xpxifqm8qcld3k-source': trying next substituter
substitution of '/nix/store/96zbdkrn93l6qjwq23xpxifqm8qcld3k-source': trying next substituter
substitution of '/nix/store/96zbdkrn93l6qjwq23xpxifqm8qcld3k-source': trying next substituter
substitution of '/nix/store/96zbdkrn93l6qjwq23xpxifqm8qcld3k-source': done
substitution of '/nix/store/96zbdkrn93l6qjwq23xpxifqm8qcld3k-source': in final_awaiter
substitution of '/nix/store/96zbdkrn93l6qjwq23xpxifqm8qcld3k-source': goal destroyed
substitution of '/nix/store/96zbdkrn93l6qjwq23xpxifqm8qcld3k-source' failed, will try to download: error: path '/nix/store/96zbdkrn93l6qjwq23xpxifqm8qcld3k-source' is required, b
ut there is no substituter that can build it
locking path "/home/figsoda/.cache/nix/fetcher-locks/ebf60d0afb4b1d979846e02870fa1b3c576268fe51fba73265dcb861723ddef4"
lock acquired on "/home/figsoda/.cache/nix/fetcher-locks/ebf60d0afb4b1d979846e02870fa1b3c576268fe51fba73265dcb861723ddef4.lock"
using cache entry 'tarball:{"url":"https://github.com/nix-community/nix-init/archive/refs/tags/v0.3.5.tar.gz"}' -> '{"etag":"W/\"992d572c7fb46e916fa1e4a3c59bca0543bdce81c36f82f5a
91fc2cbe97abeab\"","lastModified":1778805587,"treeHash":"2cd10ceeaadcf77822106c3ad03514dd0cd72f45"}'
using cache entry 'treeHashToNarHash:{"treeHash":"2cd10ceeaadcf77822106c3ad03514dd0cd72f45"}' -> '{"narHash":"sha256-D4p0wZpIkDdfB5yrZYwVe+nBBoYqA/Lxm0PF1PfBwEs="}'
using cache entry 'treeHashToNarHash:{"treeHash":"2cd10ceeaadcf77822106c3ad03514dd0cd72f45"}' -> '{"narHash":"sha256-D4p0wZpIkDdfB5yrZYwVe+nBBoYqA/Lxm0PF1PfBwEs="}'
lock released on "/home/figsoda/.cache/nix/fetcher-locks/ebf60d0afb4b1d979846e02870fa1b3c576268fe51fba73265dcb861723ddef4.lock"
got tree '«https://github.com/nix-community/nix-init/archive/refs/tags/v0.3.5.tar.gz?narHash=sha256-D4p0wZpIkDdfB5yrZYwVe%2BnBBoYqA/Lxm0PF1PfBwEs%3D»/' from 'https://github.com/n
ix-community/nix-init/archive/refs/tags/v0.3.5.tar.gz?narHash=sha256-D4p0wZpIkDdfB5yrZYwVe%2BnBBoYqA/Lxm0PF1PfBwEs%3D'
using cache entry 'sourcePathToHash:{"fingerprint":"tarball:sha256-D4p0wZpIkDdfB5yrZYwVe+nBBoYqA/Lxm0PF1PfBwEs=","method":"nar","path":"/"}' -> '{"hash":"sha256-D4p0wZpIkDdfB5yrZ
YwVe+nBBoYqA/Lxm0PF1PfBwEs="}'
source path '«https://github.com/nix-community/nix-init/archive/refs/tags/v0.3.5.tar.gz?narHash=sha256-D4p0wZpIkDdfB5yrZYwVe%2BnBBoYqA/Lxm0PF1PfBwEs%3D»/' cache hit in '/nix/stor
e/96zbdkrn93l6qjwq23xpxifqm8qcld3k-source' (hash 'sha256-D4p0wZpIkDdfB5yrZYwVe+nBBoYqA/Lxm0PF1PfBwEs=')
performing daemon worker op: 1
evaluating file '/nix/store/96zbdkrn93l6qjwq23xpxifqm8qcld3k-source/default.nix'
«lambda @ /nix/store/96zbdkrn93l6qjwq23xpxifqm8qcld3k-source/default.nix:1:1»

compared to materialized

$ nix eval -f . --option tarball-ttl 1 -vvvv
evaluating file '«nix-internal»/derivation-internal.nix'
evaluating file '/home/figsoda/cfg/default.nix'
performing daemon worker op: 10
using substituted/cached path '/nix/store/96zbdkrn93l6qjwq23xpxifqm8qcld3k-source' for 'https://github.com/nix-community/nix-init/archive/refs/tags/v0.3.5.tar.gz'
performing daemon worker op: 1
performing daemon worker op: 1
evaluating file '/nix/store/96zbdkrn93l6qjwq23xpxifqm8qcld3k-source/default.nix'
«lambda @ /nix/store/96zbdkrn93l6qjwq23xpxifqm8qcld3k-source/default.nix:1:1»

[tomberek]

Whats the difference in the setup of those two cases?

[figsoda]

they both eval the same import (fetchTarball ...), in the first one the path is not materialized (only present in the tarball cache), in the second one the path already exists in the store

[xokdvium]

Substitution is attempted unconditionally, though genitive narinfo cache would amortise this anyway. I guess the order could be swapped a bit to prioritise the tarball cache before substitution.

[tomberek]

Substitution is attempted unconditionally, though genitive narinfo cache would amortise this anyway. I guess the order could be swapped a bit to prioritise the tarball cache before substitution.

This happens in fetchers.cc. So we'd need something like moving the substitution vs cache check vs actual fetching logic to be implemented on a per-fetcher basis. Or expose a cheap isAlreadyCached method?

(edit: something like this? 7287651)

[figsoda]

btw i didnt mean to block this pr, this can totally be done in a separate pr, i was just curious about other improvements we can make