fix(metastore): do not hold the state guard across a lazy index load

[aayushbaluni]

Problem

FileBackedMetastore::index awaited LazyFileBackedIndex::get() while holding the metastore state guard. On the first access to an index that is still lazy, that await spans a full metastore.json download from storage.

tokio's RwLock is write-preferring, so a single writer queued behind that download blocks every reader arriving after it — including readers of unrelated indexes whose metadata is already resident in memory. Every index is uninitialized after a process start, so this path is reached routinely rather than exceptionally.

The same shape appeared three times in index(): the read fast path, the write-branch re-check, and the insert path.

Measured

Using @YZL0v3ZZ's reproduction from #6840 (added here as a regression test), reading an unrelated already-loaded index while a cold load is in flight and one writer is queued:

	before	after
unrelated warm read, no writer queued (baseline)	166 µs	166 µs
unrelated warm read, one writer queued	stalls — observed after 514 ms	20 consecutive reads complete, 232 ms total

Fix

Hold the lock only long enough to resolve the handle, then release it and load outside.

LazyIndexStatus::Active now carries an Arc<LazyFileBackedIndex>. That matters for correctness, not just ergonomics: LazyFileBackedIndex owns a OnceCell, so cloning the value out of the map would create a second cell and break the existing guarantee — stated in the comment right above the write branch — that concurrent callers observe a single instance and perform a single load. Cloning the Arc shares one cell, so the dedup is preserved.

get_index_mutex becomes resolve_lazy_index, which is synchronous by construction — it cannot reintroduce an await under the guard without a signature change.

In the insert path the index is handed to the constructor, so OnceCell::new_with has already populated it and resolving after dropping the guard performs no I/O. That is what makes reordering the insert ahead of the resolve safe.

Considered consequence

A concurrent delete_index can now interleave between the clone and the load, where previously it waited for the read guard. The outcome is unchanged: the caller holds an Arc it already obtained, and in the old ordering the delete simply happened immediately afterwards instead. No caller relies on the state lock being held across the load — the other LazyIndexStatus::Active sites are all Active(_) patterns that never await.

Tests

quickwit-metastore/tests/file_backed_cold_load_cascade.rs, adapted from the reporter's reproduction into a regression test: it now asserts the unrelated read completes rather than asserting it stalls.

• 123 unit tests pass, including test_file_backed_metastore_race_condition, which is the existing guard on the single-instance invariant this change touches.
• Mutation-checked: moving the read guard back out so it outlives the await fails the new test on probe 1, with the diagnostic read of unrelated index 'index-warm' stalled behind the queued writer while 'index-cold' was still loading.
• cargo clippy -p quickwit-metastore --all-targets: no warnings. cargo +nightly fmt --check: clean.

Note on how this was run

I am on Windows without a linker, so everything above ran in a rust:1.90-bookworm container; rust-toolchain.toml pinned the actual toolchain to 1.96 as usual. The container's protoc regenerates quickwit-proto/src/codegen/ with a one-character whitespace difference in a doc comment — that is a build artifact, not part of this change, and it is reverted here.

Closes #6840