refactor(core): drain tasks before graph deletion

[imbajin]

A cancelled task can still be executing or saving its completion callback. Graph deletion previously treated logical cancellation or a failed scheduler close as sufficient to remove data and registrations. This change waits for the actual local owners to exit first.

• Fence task admission in both schedulers and retain execution/cancellation owners through callback persistence and transaction cleanup. Closing a scheduler alone leaves unstarted persistent tasks in storage.
• Drain before clearing graph data or PD metadata. On a drain timeout, retain the graph and its registration with task admission closed. Retry deletion after owners exit, or restart before destructive cleanup to resume task admission; remove configuration only after successful local closure.
• Keep schedulers registered after uncertain cleanup. Explicit global shutdown still gives running workers the existing executor termination grace; it does not acknowledge graph closure. Keep long-running cron queries outside the admission monitor so shutdown can reach its timed barrier. Use initialized/actual-start/drain fixtures while preserving the precise missing-graph, task-deletion and ephemeral-absence assertions.

flowchart LR
    A[Stop task admission] --> B[Drain execution, callbacks and transaction cleanup]
    B -->|Drained| C[Delete graph data]
    B -->|Timeout| D[Keep task admission closed; retry deletion]

Loading

Scope is intentionally limited to local task shutdown and safe deletion. Stop writes and tasks on every Server before maintenance, and do not recreate the same graph name until deletion completes. Backend/metadata deletion is nontransactional; errors after destructive cleanup require operator inspection. Owner-cleanup failure does not count as a successful drain.

Hot reload recovery, cross-Server notification reconciliation, same-name replacement and task-record deletion policies are deferred to separate work. Use a restart for configuration changes. Their full source is preserved outside this PR; no global lifecycle coordinator or fallback executor is added.

Validation: formatting, clean reactor compilation and focused scheduler/deletion/permission/task/TTL regressions passed. Separate Memory and RocksDB processes verify that a real pending task persists its terminal record during global shutdown while graph data, configuration and scheduler registration remain intact. The latest head passed Linux Memory/HStore, macOS Intel/ARM and Commons/RPC suites. Complete coverage retains every baseline source file, includes RPC, and passes both coverage checks. Org test jobs also passed; its two final input-validity checks ran after the PR was closed and correctly rejected the closed PR. These are not reported as green checks.

See docs/server-runtime.md for the maintenance contract. Paired source: org #285 / ASF #3308. The broader website documentation #519 is held as a draft pending scope reconciliation and is not a merge prerequisite.

Conceptual overview

Conceptual diagram; not runtime test evidence.

[codecov]

Codecov Report

❌ Patch coverage is 62.10191% with 119 lines in your changes missing coverage. Please review.
✅ Project coverage is 52.74%. Comparing base (2691ae5) to head (337d7b6).

Files with missing lines	Patch %	Lines
...pache/hugegraph/task/DistributedTaskScheduler.java	57.06%	57 Missing and 22 partials ⚠️
...n/java/org/apache/hugegraph/core/GraphManager.java	46.66%	15 Missing and 1 partial ⚠️
...g/apache/hugegraph/task/StandardTaskScheduler.java	81.81%	7 Missing and 7 partials ⚠️
...n/java/org/apache/hugegraph/StandardHugeGraph.java	52.94%	7 Missing and 1 partial ⚠️
...in/java/org/apache/hugegraph/task/TaskManager.java	66.66%	0 Missing and 2 partials ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master    #3308      +/-   ##
============================================
+ Coverage     52.67%   52.74%   +0.07%     
- Complexity     2070     6126    +4056     
============================================
  Files           770      770              
  Lines         67694    67937     +243     
  Branches       9172     9210      +38     
============================================
+ Hits          35655    35835     +180     
- Misses        28489    28544      +55     
- Partials       3550     3558       +8     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[bitflicker64]

Blocking: yes. Summary: The deletion ordering and the retained registrations work as described, but three paths behave differently from master for callers that are not retrying a failed deletion: a drop with the local scheduler no longer waits task.wait_timeout for short tasks, a plain close of the distributed scheduler cancels dispatched tasks that never started, and reload() stops halfway when one graph cannot close. Evidence: read the full diff 87a06a9..9918ff2 (14 files, +3653/-169) and the surrounding code in GraphManager, StandardHugeGraph, TaskManager, HugeTask and both schedulers; built both trees with JDK 17 (mvn -pl hugegraph-server/hugegraph-test -am install -DskipTests) and ran one JUnit probe on each: dropping a memory graph while a 1.5 s task runs returns after 1720 ms on 87a06a9 and throws after 0 ms on 9918ff2; the scheduler-close and reload() points are from reading the code and from the assertions in DistributedTaskAdmissionTest, not from a run. All check runs at 9918ff2 are green or skipped.

[imbajin]

The scope has been narrowed to local task admission, physical execution/callback drain, and safe graph deletion. Global lifecycle coordination, hot reload restoration, cross-Server notification reconciliation, task-record deletion recovery, and fallback executors have been removed from this PR and recorded for separate work.

Previous descriptions claiming those broader guarantees are superseded. Configuration changes require a restart while reload recovery is deferred; all Servers must stop writes and tasks before deletion. The broader website documentation PRs are now drafts pending reconciliation.

Core regression tests and clean compilation passed. The missing-graph fixture now uses normal GraphManager construction and still requires the original IllegalArgumentException. Current-head Linux CI and complete coverage are pending.

[bitflicker64]

Blocking: yes. Summary: The drain-before-delete ordering and the retained registrations hold up on reading, but two effects reach callers that are not retrying a deletion: TaskManager.shutdown() now throws before stopping any executor when a scheduler is still registered, and a refused drop leaves the local scheduler closed for good on a graph that is kept. Evidence: read the full diff 68fb045..bb3ebc8 (14 files, +1789/-146) and the surrounding code in GraphManager, StandardHugeGraph, TaskManager, HugeTask, HugeFactory, HugeGraphServer and both schedulers; both findings are static, from the head source and the assertions in StandardTaskAdmissionTest and HugeFactoryTest, and no test was run for this review. All check runs at bb3ebc8 are green or skipped.

[imbajin]

Source cross-check of the current memory CI error: ConsumersTest.testAwaitThrowsWhenConsumerThrows calls provide(1) before entering the try/catch around await(). If the consumer throws immediately after the offer, provide() observes stopped and rethrows that original exception first. This matches the uncaught worker exception in this run.

Both Consumers.java and ConsumersTest.java are byte-identical in base 68fb045, bb3ebc8, d61a8cd and the tested merge. This is a source-supported diagnosis of an existing fixture race, not a reproduced regression introduced by the shutdown correction. A focused fix is to gate the consumer's throw until provide() returns, then assert the intended await() failure. Please retain the exception assertion rather than weakening it or treating a retry alone as proof. No new test was run for this review.

[imbajin]

Addressed the Memory CI fixture race in the four-line test correction. The existing gate now holds the injected consumer failure until provide() returns. The original await() RuntimeException/message assertions, one-second test timeout and executor cleanup are unchanged; production Consumers is unchanged.

A controlled real-consumer post-offer interleaving reproduced the old failure escaping from provide(), then confirmed the gated version reports that same exception from await(). All ConsumersTest cases, formatting and the clean reactor compile passed. The diagnostic queue observer is external evidence only. The branch is rebased onto current ASF master; Java/POM/proto/config sources match the tested version. Latest-head Linux CI and full coverage are still pending.