[Core] Actor reorder timer is re-armed on every task arrival, so a missing seq_no is never skipped

[LuciferYang]

What happened + What you expected to happen

An ordered actor task queue arms one reorder timer per concurrency group when the head of the queue is not the seq_no it is waiting for. The timer is what eventually skips a seq_no that never arrives, so the actor does not wait on it forever. ExecuteQueuedTasks re-arms that timer with a full reorder_wait_seconds_ on every pass:

https://github.com/ray-project/ray/blob/master/src/ray/core_worker/task_execution/ordered_actor_task_execution_queue.cc#L282-L289

and it runs on every task arrival (end of EnqueueTask) and again when a dependency resolves. So a client submitting faster than the timeout keeps pushing the deadline out, the gap is never skipped, and the group stops making progress.

It is worse than per-group, because the loop covers every group of that caller:

https://github.com/ray-project/ray/blob/master/src/ray/core_worker/task_execution/ordered_actor_task_execution_queue.cc#L219-L221

A task arriving in group io pushes out the deadline of a blocked group default. That is also what makes the stall unbounded in practice: with a single group the caller's in-flight window (kMaxBytesInFlight, 16 MiB) eventually stops it sending, arrivals cease and the timer fires; with a second group still completing and replying, the caller never blocks and the arrivals never stop.

Nothing is logged above DEBUG while this happens. The queue logs the enqueue at DEBUG, the INFO line needs client_processed_up_to >= next_seq_no and the ERROR line needs a stale seq_no, so neither is reached. The only outward sign is tasks sitting in PENDING_ACTOR_TASK_ORDERING_OR_CONCURRENCY on the dashboard.

Expected: the deadline runs from when the group first blocked on the missing seq_no, so it expires and the gap is skipped.

Versions / Dependencies

master (0e488164f8). The timeout is actor_scheduling_queue_max_reorder_wait_seconds, default 30s.

Reproduction script

A unit-level repro is the clearest, since it needs a seq_no that never arrives:

// reorder_wait_seconds = 1
EnqueueWithFetch(queue, waiter, 0, -1, MakeTaskToExecute(task_spec));  // runs
EnqueueWithFetch(queue, waiter, 2, -1, MakeTaskToExecute(task_spec));  // blocks on 1
// then keep enqueueing 3, 4, 5... every 100ms, pumping io_service in between.
// n_canceled stays 0 for as long as tasks keep arriving; the gap at 1 is never skipped.

From Python it takes a client that submits to an actor faster than 30s per task while one request is lost or delayed — the loss is the hard part to stage, which is why the timer exists in the first place.

Issue Severity

Medium: it needs a lost or long-delayed request to start, but once started the group is stuck for as long as the client keeps submitting, and there is no log line pointing at it.

AI assistance

AI assistance was used to investigate and write this up. The code references were verified against master.