[RFC]: Deterministic fault-injection harness for disaggregated serving

[sylvesterkaczmarek]

Motivation

TensorRT-LLM's disaggregated serving path now spans several asynchronous reliability boundaries: context/generation orchestration, KV-cache transfer, completion notifications, cancellation/timeouts, cluster storage, and worker lifecycle. The repository has good focused regression tests for individual failures, but each new failure mode currently tends to require a bespoke test harness.

I would like to propose a reusable, deterministic fault-injection harness for disaggregated serving so reliability behavior can be exercised systematically rather than one incident at a time.

I searched the current docs/issues for an equivalent fault-injection or chaos-testing framework and did not find one. Existing tests such as bounded transceiver polling and in-flight cancellation cover specific cases, but not a shared failure-model layer.

Proposed Change

Add an opt-in testing framework that can inject failures at defined disaggregated-serving boundaries while keeping normal production behavior unchanged.

A first version could support scripted faults such as:

• delay or drop KV-transfer completion notifications;
• force transfer timeout/cancellation at a chosen lifecycle point;
• fail a transfer before start, while in flight, or after data movement but before acknowledgement;
• return stale, expired, missing, or delayed cluster-storage state;
• inject duplicate/out-of-order watch events;
• simulate context or generation worker disconnects;
• cancel a request while transfer/storage work is outstanding.

Conceptually, tests could define a deterministic scenario rather than custom mocks for every case:

scenario = DisaggFaultScenario([
    drop("kv_transfer_complete", request="r1"),
    advance_time(ms=5000),
])

with disagg_faults(scenario) as faults:
    result = run_test_request(...)

faults.assert_consumed()

The harness should record a compact lifecycle trace so tests can assert invariants such as:

• bounded completion or bounded failure;
• no leaked transfer/request state;
• cancellation releases owned resources;
• stale cluster-storage data is not surfaced as live;
• retries/reconciliation do not duplicate completion;
• unaffected requests continue progressing.

I would start with a CPU-only deterministic layer around the existing Python disaggregation/transceiver/storage boundaries, using fake time and fake transports where possible. A later phase could add a small loopback/integration matrix for real NIXL/UCX-capable environments without making GPU/fabric tests a prerequisite for the core harness.

The intent is test infrastructure, not a production chaos-injection API. Production code would only gain narrowly scoped injection seams where the current architecture does not already expose a testable boundary.

If maintainers agree with the direction, I can prepare the implementation as several small PRs rather than one large change: core scenario/fault API, transceiver integration, storage/watch integration, then representative disaggregated regression migrations.

Feedback Period

1-2 weeks would be ideal before implementation.

Any Other Things

This proposal is motivated by the same reliability area where recent regressions have appeared around KV-transfer timeout/cancellation and cluster-storage semantics. The goal is to make future fixes easier to prove across a consistent failure matrix, not to change the serving contract itself.

Before submitting a new issue...

• Searched relevant issues, documentation, and existing disaggregated-serving tests.