[#13777] feat(tms): In-job S0 gates, OccupancyGate, and REST Feature lifecycle by lasdf1234 · Pull Request #13841 · apache/gravitino

lasdf1234 · 2026-10-14T06:12:06Z

What changes were proposed in this pull request?

Combines in-job gates and OccupancyGate / REST Feature work on top of merged #13774 (formerly tracked as #13816, folded into #13777):

In-job gates

Package maintenance.injob: S0 sample → snapshot-drift → recommend gates
TableMaintenanceInJobSampler / TableMaintenanceInJobPipeline (gateCronJob / gateCommitChainType)
InJobRecommender + RecommendDecision (injected in tests; no production placeholder recommender)
Persist sampled metrics via idempotent insertMetrics (first-writer-wins on unique race) and set before_snapshot_id (same-S0 retry safe)
Unit tests bind soft-slot rows with OccupancyClaim (insertInFlight + bindJobId)

OccupancyGate + REST Feature

OccupancyGate over soft-slot occupancy: unbound TTL evaluated on the database clock (hasLiveUnboundClaim); bound rows consult JobMetaService with no finished_at backfill
TableMaintenanceRESTFeature starts the three db-scheduler pools when the TMS Jersey extension package is enabled, and stops them on Jersey DESTROY_FINISHED (before entity-store DataSource close); configure-after-stop re-registers the destroy listener
Jersey / server-common dependencies for the Feature SPI
Unit tests for OccupancyGate (including DB-clock unbound expiry) and Feature destroy listener

Out of scope: Four maintenance type executors, OCC commit-chain, PolicyExpand / Spark submit / IRC listener pipelines.

Why are the changes needed?

In-job S0/drift/recommend gates and control-plane occupancy / Feature lifecycle for Epic #13760.

Fix: #13777

Does this PR introduce any user-facing change?

No public REST resources yet. Enabling the TMS extension package starts in-process scheduler pools via TableMaintenanceRESTFeature.

How was this patch tested?

./gradlew :plugins:tms:test --tests 'org.apache.gravitino.maintenance.injob.*' \
  --tests 'org.apache.gravitino.maintenance.occupancy.*' \
  --tests 'org.apache.gravitino.maintenance.web.rest.feature.*' \
  --tests 'org.apache.gravitino.maintenance.storage.service.*' -PskipITs

…mend gate Add the Spark-side sample → drift → recommend pipeline for cron and commit-chain types without executing the four maintenance operations. Co-authored-by: Cursor <cursoragent@cursor.com>

… upsert Move package to maintenance.injob, drop FileCountThresholdRecommender from main, rename gate APIs/outcomes, and putMetrics on re-sample. Co-authored-by: Cursor <cursoragent@cursor.com>

Re-sampling the same snapshot is out of scope; keep insertMetrics only. Co-authored-by: Cursor <cursoragent@cursor.com>

…-slot APIs Rebase onto feature/tms-scheduler-13773 and replace JobMetaService insertJob with insertInFlight + bindJobId(created_at fence). Co-authored-by: Cursor <cursoragent@cursor.com>

…eature Wire OccupancyGate over soft-slot claim/reclaim/bind/finish and start/stop the three db-scheduler pools from TableMaintenanceRESTFeature. Co-authored-by: Cursor <cursoragent@cursor.com>

…Claim APIs Add Jersey/server-common deps for TableMaintenanceRESTFeature and update tests to bind via OccupancyClaim after apache#13774 landed on main. Co-authored-by: Cursor <cursoragent@cursor.com>

roryqi

I found two lifecycle/clock-consistency issues that should be addressed before relying on the new TMS scheduling path.

roryqi · 2026-10-14T08:41:41Z

+    DataSource dataSource = resolveEntityStoreDataSource();
+    TableMaintenanceSchedulerPools pools = new TableMaintenanceSchedulerPools();
+    pools.start(dataSource, TableMaintenanceConfig.from(config));
+    SCHEDULER_POOLS.set(pools);


stopSchedulers() is not wired into the Jersey/Jetty or GravitinoServer.stop() lifecycle. The db-scheduler shutdown hooks only run when the JVM terminates, so stopping a server in the same JVM closes the relational DataSource while these pools remain active. A later configure() also starts another set and overwrites this reference, leaving the old pools unreachable. Please register a container shutdown callback that stops these pools before the entity store closes, and guard repeated configuration.

Addressed in ef8ec42:

Registered a Jersey ApplicationEventListener that calls stopSchedulers() on DESTROY_FINISHED (runs during Jetty stop, before GravitinoEnv closes the entity-store DataSource).

Guarded repeated configure() so we do not overwrite a live pool reference / leak the previous pools.

roryqi · 2026-10-14T08:41:41Z

+   * @return true when any open row is a live unbound claim or an active Gravitino job
+   */
+  public boolean isInFlight(long tableId, long policyId) {
+    return isInFlight(tableId, policyId, System.currentTimeMillis());


created_at is deliberately written and reclaimed using the database clock, but this default path compares it with the local JVM clock. In a multi-node deployment, clock skew can make a fresh claim appear expired or keep an expired claim live too long, so shouldSkip() disagrees with the DAO's DB-clock-based acquire/reclaim decision. Please evaluate unbound-claim TTL in the DAO using the database clock (or otherwise use one shared clock), and cover skew in a test.

Addressed in ef8ec42:

Unbound TTL is now evaluated in the DAO via hasLiveUnboundClaim / selectLiveUnboundByTableAndPolicy against the database clock (same timeline as acquire/reclaim).

OccupancyGate.isInFlight no longer subtracts JVM now from created_at.

Added testExpiredUnboundClaimIsNotInFlightOnDbClock so gate occupancy agrees with DB-clock expiry / reclaim.

…lock unbound TTL Register an ApplicationEventListener so pools stop before the entity store closes, skip duplicate configure, and evaluate live unbound claims in the DAO against the database clock so OccupancyGate agrees with acquire/reclaim under JVM skew. Co-authored-by: Cursor <cursoragent@cursor.com>

…s insert race Allow configure-after-stop to re-register the destroy listener; make recordBeforeSnapshot idempotent for the same S0; treat metrics unique conflicts as first-writer-wins. Co-authored-by: Cursor <cursoragent@cursor.com>

github-actions · 2026-10-14T16:02:19Z

Code Coverage Report

Overall Project	72.02% `+0.16%`	🟢
Files changed	87.33%	🟢

Module	Coverage
aliyun	19.74%	🔴
api	54.68%	🟢
authorization-common	85.96%	🟢
authorization-ranger	14.44%	🔴
aws	53.54%	🟢
aws-bundle	0.0%	🔴
azure	32.1%	🔴
azure-bundle	0.0%	🔴
catalog-common	29.49%	🔴
catalog-fileset	82.37%	🟢
catalog-glue	73.24%	🟢
catalog-hive	83.63%	🟢
catalog-jdbc-common	57.62%	🟢
catalog-jdbc-doris	87.56%	🟢
catalog-jdbc-mysql	81.91%	🟢
catalog-jdbc-postgresql	84.02%	🟢
catalog-jdbc-starrocks	79.16%	🟢
catalog-kafka	75.21%	🟢
catalog-lakehouse-generic	80.67%	🟢
catalog-lakehouse-hudi	79.1%	🟢
catalog-lakehouse-iceberg	86.0%	🟢
catalog-lakehouse-paimon	84.36%	🟢
catalog-model	77.99%	🟢
cli	44.62%	🟢
client-java	79.32%	🟢
client-java-runtime	0.0%	🔴
common	60.49%	🟢
core	87.43% `-0.02%`	🟢
filesystem-hadoop3	76.89%	🟢
flink	55.36%	🟢
flink-common	32.2%	🔴
flink-runtime	0.0%	🔴
gcp	32.2%	🔴
hadoop-auth	68.0%	🟢
hadoop-common	17.84%	🔴
hive-metastore-common	62.3%	🟢
hive-metastore3-libs	58.8%	🟢
iceberg-aliyun-bundle	0.0%	🔴
iceberg-common	66.53%	🟢
iceberg-rest-server	77.01%	🟢
idp-basic	86.77%	🟢
integration-test-common	0.0%	🔴
jobs	70.23%	🟢
k8s-job-executor	99.81%	🟢
lance-common	39.9%	🔴
lance-rest-server	69.51%	🟢
lineage	59.39%	🟢
optimizer	84.27%	🟢
optimizer-api	40.87%	🟢
server	91.43%	🟢
server-common	84.04% `+2.2%`	🟢
spark	61.79%	🟢
tencent	81.78%	🟢
tms	80.0% `-0.68%`	🟢
trino-connector	63.31%	🟢

Files

Module	File	Coverage
core	AuthorizationRequestContext.java	80.95%	🟢
server-common	JcasbinLoadedRolesCache.java	96.3%	🟢
	JcasbinAuthorizer.java	90.58%	🟢
tms	InJobGateOutcome.java	100.0%	🟢
	RecommendDecision.java	100.0%	🟢
	TableMaintenanceInJobPipeline.java	95.65%	🟢
	TableMaintenanceActionMetaService.java	94.79%	🟢
	TableMaintenanceActionBaseSQLProvider.java	94.74%	🟢
	OccupancyGate.java	82.46%	🟢
	TableMaintenanceInJobSampler.java	80.95%	🟢
	TableSnapshotMetricsMetaService.java	56.67%	🔴
	TableMaintenanceRESTFeature.java	26.32%	🔴
	InJobRecommender.java	0.0%	🔴
	SnapshotHead.java	0.0%	🔴

lasdf1234 and others added 6 commits October 14, 2026 14:10

[apache#13777] feat(tms): In-job S0 sample, snapshot drift, and recom…

0515e56

…mend gate Add the Spark-side sample → drift → recommend pipeline for cron and commit-chain types without executing the four maintenance operations. Co-authored-by: Cursor <cursoragent@cursor.com>

[apache#13777] refactor(tms): Clean up in-job gate naming and metrics…

540646e

… upsert Move package to maintenance.injob, drop FileCountThresholdRecommender from main, rename gate APIs/outcomes, and putMetrics on re-sample. Co-authored-by: Cursor <cursoragent@cursor.com>

[apache#13777] refactor(tms): Drop metrics upsert; S0 is insert-once

dce7f99

Re-sampling the same snapshot is out of scope; keep insertMetrics only. Co-authored-by: Cursor <cursoragent@cursor.com>

[apache#13777] fix(tms): Adapt in-job gates to ActionMetaService soft…

f778359

…-slot APIs Rebase onto feature/tms-scheduler-13773 and replace JobMetaService insertJob with insertInFlight + bindJobId(created_at fence). Co-authored-by: Cursor <cursoragent@cursor.com>

[apache#13816] feat(tms): Add OccupancyGate and TableMaintenanceRESTF…

d3e7def

…eature Wire OccupancyGate over soft-slot claim/reclaim/bind/finish and start/stop the three db-scheduler pools from TableMaintenanceRESTFeature. Co-authored-by: Cursor <cursoragent@cursor.com>

[apache#13816] fix(tms): Align OccupancyGate with soft-slot Occupancy…

88f0e6b

…Claim APIs Add Jersey/server-common deps for TableMaintenanceRESTFeature and update tests to bind via OccupancyClaim after apache#13774 landed on main. Co-authored-by: Cursor <cursoragent@cursor.com>

lasdf1234 changed the title ~~[#13777] feat(tms): In-job S0 sample, snapshot drift, and recommend gate~~ [#13777][#13816] feat(tms): In-job gates + OccupancyGate and REST Feature lifecycle Oct 14, 2026

This was referenced Oct 14, 2026

[Subtask] TMS: OccupancyGate and REST Feature lifecycle #13816

Closed

[Subtask] TMS: In-job S0/drift/recommend gates and OccupancyGate REST Feature #13777

Closed

[EPIC] TMS: Persistence, scheduler, and in-job Spark path #13760

Open

lasdf1234 self-assigned this Oct 14, 2026

lasdf1234 changed the title ~~[#13777][#13816] feat(tms): In-job gates + OccupancyGate and REST Feature lifecycle~~ [#13777][#13816] feat(tms): In-job S0/drift/recommend gates and OccupancyGate REST Feature Oct 14, 2026

lasdf1234 requested a review from roryqi October 14, 2026 06:31

roryqi reviewed Oct 14, 2026

View reviewed changes

lasdf1234 and others added 2 commits October 14, 2026 16:54

lasdf1234 changed the title ~~[#13777][#13816] feat(tms): In-job S0/drift/recommend gates and OccupancyGate REST Feature~~ [#13777][#13816] feat(tms): In-job S0 gates, OccupancyGate, and REST Feature lifecycle Oct 14, 2026

lasdf1234 changed the title ~~[#13777][#13816] feat(tms): In-job S0 gates, OccupancyGate, and REST Feature lifecycle~~ [#13777] feat(tms): In-job S0 gates, OccupancyGate, and REST Feature lifecycle Oct 14, 2026

roryqi approved these changes Oct 14, 2026

View reviewed changes

roryqi merged commit 189ba4d into apache:main Oct 15, 2026
35 checks passed

Navigation Menu

Repository navigation

[#13777] feat(tms): In-job S0 gates, OccupancyGate, and REST Feature lifecycle - #13841

[#13777] feat(tms): In-job S0 gates, OccupancyGate, and REST Feature lifecycle#13841
roryqi merged 8 commits into
apache:mainfrom
lasdf1234:feature/tms-injob-sample-13760

lasdf1234 commented Oct 14, 2026 •

edited

Loading

Uh oh!

roryqi left a comment

Uh oh!

roryqi Oct 14, 2026

Uh oh!

lasdf1234 Oct 14, 2026

Uh oh!

roryqi Oct 14, 2026

Uh oh!

lasdf1234 Oct 14, 2026

Uh oh!

github-actions Bot commented Oct 14, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

lasdf1234 commented Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

roryqi left a comment

Choose a reason for hiding this comment

Uh oh!

roryqi Oct 14, 2026

Choose a reason for hiding this comment

Uh oh!

lasdf1234 Oct 14, 2026

Choose a reason for hiding this comment

Uh oh!

roryqi Oct 14, 2026

Choose a reason for hiding this comment

Uh oh!

lasdf1234 Oct 14, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Oct 14, 2026

Code Coverage Report

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

lasdf1234 commented Oct 14, 2026 •

edited

Loading