fix(control-plane): do not hold the create-index reply for the plan rebuild

[aayushbaluni]

Proposal, not a decision

#6841 offers three directions and the choice is yours. This implements the first one — reply once the metastore write has completed and let the rebuild proceed asynchronously — because I argued for it on the issue and it is the only one that does not touch the apply path.

I am opening it as a concrete thing to react to rather than waiting. Close it freely if you prefer bounding the reply wait, or extending apply deadlines to Ready indexers, and I will implement whichever you pick instead.

What was happening

create_index moved its reply into a task awaiting the next plan rebuild whenever the new index carried a source:

if should_rebuild_plan {
    let rebuild_plan_notifier = self.rebuild_plan_debounced(ctx);
    tokio::task::spawn(async move {
        rebuild_plan_notifier.await;
        reply(Ok(response));
    });
}

The rebuild's completion is signalled by a generation counter, and that counter only advances once every ApplyIndexingPlan RPC of the rebuild has finished — NotifyChangeOnDrop publishes the generation on Drop (indexing_scheduler/change_tracker.rs), and each apply task holds a clone.

Apply RPCs to Ready indexers carry no deadline, deliberately — indexing_scheduler/mod.rs bounds only the draining states and says so: "Ready indexers get no timeout." So one Ready indexer that accepts its apply and never answers keeps the generation from advancing, and the caller never gets a reply for an index already committed to the metastore. A retry returns AlreadyExists, and since that indexer joins every later rebuild, subsequent creates queue behind it too.

The change

Trigger the rebuild without coupling the reply to it. rebuild_plan_debounced already separates those concerns — it sends RebuildPlan through the debouncer and returns a waiter — and the waiter is only a cloned watch::Receiver, so dropping it cannot affect the rebuild.

This does not change what a caller can observe. self.model.add_index runs before the reply on both branches already, so a request issued after the response (opening shards, for instance) still sees the index and its sources. The rebuild governs when indexers are told to run pipelines, which was never part of this response.

I deliberately left the Ready-indexer deadline alone. That asymmetry is recorded in a comment, so reversing it is a semantics decision rather than a gap to fill.

Verification, and what is missing from it

cargo test -p quickwit-control-plane: 150 passed, 0 failed. cargo clippy -p quickwit-control-plane --all-targets and cargo +nightly fmt --check are both clean.

Two existing tests already exercise this path with a source — test_control_plane_handles_rebalance_shards_callback and test_control_plane_get_debug_info both create_index with an ingest_v2 source and then immediately open shards. They still pass, which is the evidence that replying earlier does not break a caller that needs the index visible straight away.

I could not write a regression test that fails on the old code, and I would rather say so than ship one that looks like it does. I tried two shapes. An indexer backed by an undrained test mailbox does not work: the actor ask carries a timeout that Universe::with_accelerated_time fires immediately, so the apply completes with an error and the generation advances. Replacing it with a direct IndexingService impl whose apply_indexing_plan is std::future::pending().await also still passed on the unfixed code, which I take to mean no apply was dispatched to it at all — with a fresh index and no open shards the plan for that indexer does not change, so the scheduler skips applying and the notifier resolves regardless.

Getting a faithful test needs the scheduler to actually dispatch an apply, which needs more setup than I could validate confidently. If you know the shortest way to force that, I will add the test.

Addresses #6841