Skip to content

[#13800] feat(policy): Maintenance execution peers, BuiltInType registry, and minIntervalMs defaults - #13801

Closed
lasdf1234 wants to merge 20 commits into
apache:mainfrom
lasdf1234:feature/tms-policy-peers-13800
Closed

lasdf1234 wants to merge 20 commits into
apache:mainfrom
lasdf1234:feature/tms-policy-peers-13800

Conversation

@lasdf1234

@lasdf1234 lasdf1234 commented Oct 13, 2026 •

Copy link
Copy Markdown
Collaborator

What changes were proposed in this pull request?

Domain + DTO slice of Epic #13764 / #13800.

  • Carry TMS execution peers on AbstractExecutablePolicyContent (not universal PolicyContent):
    • timeTrigger — cron string (syntax not validated in api)
    • commitTrigger — boolean
    • minIntervalMs, jobOptions, updaterOptions
  • Four Iceberg maintenance contents extend the abstract type; attach peers via construction-time PolicyExecutionSpec
  • Shared peer checks in AbstractExecutablePolicyContent.validate(); universal PolicyContent.validate() only checks supportedObjectTypes
  • Orphan content rejects commitTrigger == true
  • MaintenancePolicyDefinitions registry keyed by Policy.BuiltInType for per-type minIntervalMs server defaults; resolve() rejects negative policy values
  • Iceberg PolicyContentDTO + DTOConverters round-trip peers so REST alter does not drop them
  • Design doc §5.0: policy/BuiltInType identity vs scheduler task kinds (policy-expand / cron-job / commit-job)

Out of scope (follow-ups):

Why are the changes needed?

TMS needs shared trigger / cooldown / option-bag concepts without freezing them onto every PolicyContent. Server defaults cover omitted minIntervalMs. BuiltInType stays the single maintenance identity (no parallel task-type enum).

Fix: #13800

Does this PR introduce any user-facing change?

Yes.

New optional server config keys (with defaults):

  • gravitino.maintenance.compaction.minIntervalMs (3600000)
  • gravitino.maintenance.manifest-rewrite.minIntervalMs (3600000)
  • gravitino.maintenance.snapshot-expiry.minIntervalMs (3600000)
  • gravitino.maintenance.orphan-cleanup.minIntervalMs (86400000)

Optional JSON peer fields on Iceberg maintenance policy contents (REST/DTO):

{
  "timeTrigger": "0 0 2 * * ?",
  "commitTrigger": true,
  "minIntervalMs": 3600000,
  "jobOptions": {},
  "updaterOptions": {}
}

How was this patch tested?

./gradlew :api:test --tests 'org.apache.gravitino.policy.*' \
  :common:test --tests 'org.apache.gravitino.dto.policy.*' \
  :core:test --tests 'org.apache.gravitino.maintenance.policy.TestMaintenanceMinIntervalConfigs' \
  -PskipITs

@lasdf1234 lasdf1234 self-assigned this Oct 13, 2026
@lasdf1234
lasdf1234 force-pushed the feature/tms-policy-peers-13800 branch from 9c48d64 to 33f296b Compare October 13, 2026 02:28
@lasdf1234 lasdf1234 changed the title [#13800] feat(policy): Attach TMS peer fields on table-maintenance content [#13800] feat(policy): Add TableMaintenanceContent peers and validation Oct 13, 2026
@lasdf1234
lasdf1234 force-pushed the feature/tms-policy-peers-13800 branch 3 times, most recently from 1c6985e to f7e2976 Compare October 13, 2026 03:01
@lasdf1234 lasdf1234 changed the title [#13800] feat(policy): Add TableMaintenanceContent peers and validation [#13800] feat(policy): Add TableMaintenanceContent peers on maintenance contents Oct 13, 2026
@lasdf1234
lasdf1234 force-pushed the feature/tms-policy-peers-13800 branch from f7e2976 to ea6e1ad Compare October 13, 2026 03:22
@lasdf1234 lasdf1234 changed the title [#13800] feat(policy): Add TableMaintenanceContent peers on maintenance contents [#13800] feat(policy): TMS MaintenancePeerFields domain model Oct 13, 2026
@lasdf1234 lasdf1234 changed the title [#13800] feat(policy): TMS MaintenancePeerFields domain model [#13800] feat(policy): TMS peer fields on policy content (domain + REST) Oct 13, 2026
@lasdf1234
lasdf1234 force-pushed the feature/tms-policy-peers-13800 branch from 7f3a26c to 04e8141 Compare October 13, 2026 07:47
@lasdf1234 lasdf1234 changed the title [#13800] feat(policy): TMS peer fields on policy content (domain + REST) [#13800] feat(policy): Add TableMaintenanceContent peers on Iceberg contents Oct 13, 2026
@lasdf1234
lasdf1234 force-pushed the feature/tms-policy-peers-13800 branch from 04e8141 to e8f52ec Compare October 13, 2026 07:49
@lasdf1234 lasdf1234 changed the title [#13800] feat(policy): Add TableMaintenanceContent peers on Iceberg contents [#13800] feat(policy): Add TMS fields on table-maintenance policy content Oct 13, 2026
…cy content

Add TableMaintenanceContent with schedule, minIntervalMs, jobOptions, and
updateOptions; wire the four Iceberg maintenance contents and PolicyContents
factories; validate via TableMaintenancePolicies on PolicyEntity.

Co-authored-by: Cursor <cursoragent@cursor.com>
lasdf1234 and others added 2 commits October 13, 2026 17:47
api javadoc cannot resolve core's PolicyEntity; use {@code} instead so
:api:javadoc passes in CI.

Co-authored-by: Cursor <cursoragent@cursor.com>
…hedule triggers

Move timeTrigger/eventTrigger/minIntervalMs/jobOptions/updateOptions onto
PolicyContent as shared concepts, and drop the combined schedule object.

Co-authored-by: Cursor <cursoragent@cursor.com>
@lasdf1234 lasdf1234 changed the title [#13800] feat(policy): Add TMS fields on table-maintenance policy content [#13800] feat(policy): Add shared policy execution peers (triggers + options) Oct 13, 2026
lasdf1234 and others added 2 commits October 13, 2026 20:43
Keep shared TimeTrigger/EventTrigger/PolicyExecutionFields on PolicyContent;
remove TableMaintenanceContent, TaskType, and Policies without wiring peers
onto Iceberg contents yet.

Co-authored-by: Cursor <cursoragent@cursor.com>
…erver defaults

Register gravitino.maintenance.task.*.minIntervalMs ConfigEntries with 1h/1d
defaults so cooldown can be read from gravitino.conf without TaskType.

Co-authored-by: Cursor <cursoragent@cursor.com>
@lasdf1234 lasdf1234 changed the title [#13800] feat(policy): Add shared policy execution peers (triggers + options) [#13800] feat(policy): Add shared execution peers and minIntervalMs defaults Oct 13, 2026
}

public Map<String, String> jobOptions() {
return jobOptions == null ? Collections.emptyMap() : jobOptions;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

jobOptions() returns the DTO's mutable backing map, unlike rewriteOptions() above, which returns an unmodifiable defensive copy. A caller can therefore mutate the DTO after validation and also change its Lombok-generated equality/hash state. Please return an immutable defensive copy here and for updateOptions() in all four DTOs. Centralizing these fields in an immutable execution-spec DTO would also remove this repetition.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed — all four Iceberg execution DTOs now return PolicyExecutionFields.copyOptions(...) for jobOptions() / updateOptions() (unmodifiable defensive copies, same idea as rewriteOptions()). Added a regression test that mutating the returned maps throws.

}

@Nullable
public String timeTrigger() {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These five new public accessors have no Javadocs, and the same pattern is repeated in the other execution-capable DTOs. Please document the new public API methods (including null/default semantics), or centralize them behind the proposed execution-spec abstraction.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed — centralized docs on PolicyContentDTO.ExecutionPeersDTO (null/empty semantics for the five peers). The four Iceberg DTOs implement that interface so the public accessors share one Javadoc contract.

@roryqi roryqi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Additional review notes:

  • The design document still defines triggers under policy_version_info.content.schedule and shows schedule.onCommit / schedule.crontab, while this implementation's wire contract is flat commitTrigger / timeTrigger. Please make the design document and the actual JSON contract consistent.
  • The same section says automated maintenance must have at least one trigger and that create/alter rejects credential-shaped option keys. Neither rule is enforced by this PR. If those are intentionally deferred, the document should clearly mark them as future behavior rather than describing them as current guarantees.
  • Please also make the cron syntax/example consistent with the scheduler contract when that validation is wired in.

The main blocking concern is the mutable, unchecked withExecutionFields API called out inline. An immutable PolicyExecutionSpec composed into executable policy contents would address that cleanly.

lasdf1234 and others added 3 commits October 14, 2026 20:51
…ecutable peers

Replace mutable withExecutionFields with construction-time PolicyExecutionSpec
on ExecutablePolicyContent. Option map values are omitted from toString/DTO
ToString to avoid leaking credentials in logs.

Co-authored-by: Cursor <cursoragent@cursor.com>
Return unmodifiable option-map copies, document peers via ExecutionPeersDTO,
and cover mutation attempts in TestPolicyDTO.

Co-authored-by: Cursor <cursoragent@cursor.com>
Use commitTrigger/timeTrigger instead of content.schedule.*; drop the
required-trigger rule; keep credential-key rejection as design intent.

Co-authored-by: Cursor <cursoragent@cursor.com>
@lasdf1234

Copy link
Copy Markdown
Collaborator Author

Thanks for the additional review notes — status:

  1. Blocking withExecutionFields — done in recent pushes: immutable PolicyExecutionSpec + ExecutablePolicyContent.executionSpec(), construction-time only; option values omitted from toString / DTO @ToString; DTO option maps return unmodifiable copies + ExecutionPeersDTO Javadoc.

  2. Design doc vs wire contract — updated design-docs/table-maintenance-service-design.md to flat peers (commitTrigger / timeTrigger / minIntervalMs / jobOptions / …), not content.schedule.*. Cron examples note that syntax is owned by the TMS / db-scheduler registration path.

  3. “At least one trigger” — removed from the design. Triggers are optional: a policy may set commitTrigger, timeTrigger, both, or neither. We do not want create/alter to require a trigger; no-trigger policies simply are not enrolled in automated expand/commit until a trigger is added.

  4. Credential-shaped option-key rejection — still the design intent (secrets must not live in policy content; create/alter should reject credential-shaped keys in option bags). Not implemented in this PR; that enforcement lands in a follow-up on the server create/alter boundary. Auth stays cluster-static in gravitino.conf in the meantime.

Keep flat commitTrigger/timeTrigger peers; adopt scheduled_*_tasks wording
from main.

Co-authored-by: Cursor <cursoragent@cursor.com>
@lasdf1234
lasdf1234 requested a review from roryqi October 14, 2026 13:04
private final Map<String, String> jobOptions;
private final Map<String, String> updateOptions;

private PolicyExecutionSpec(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PolicyExecutionSpec is now serialized as the executionSpec field of every executable policy content, but this type has only a private constructor and no Jackson creator/default constructor. POConverters.fromPolicyPO() therefore cannot read the stored content back; the current :core:test and Maintenance IT failures both report Cannot construct instance of PolicyExecutionSpec. Please make the persisted form deserializable (for example with an appropriate @JsonCreator/property mapping or a custom persistence representation) and add a POConverters round-trip test for maintenance policy content.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed — went with @JsonCreator on PolicyExecutionSpec.of(...) plus @JsonUnwrapped on AbstractExecutablePolicyContent.executionSpec, so peers stay flat on the wire (timeTrigger / commitTrigger / …) and POConverters.fromPolicyPO() can rebuild them.

Also added TestPOConverters.testFromPolicyPOIcebergCompactionWithExecutionPeers to cover the persistence round-trip (and assert the JSON is not nested under executionSpec).

…izable for POConverters

Add @jsoncreator on PolicyExecutionSpec.of and @JsonUnwrapped on the content
field so flat peer keys round-trip through JsonUtils.anyFieldMapper.

Co-authored-by: Cursor <cursoragent@cursor.com>
@lasdf1234
lasdf1234 requested a review from roryqi October 15, 2026 07:08
*/
public abstract class AbstractExecutablePolicyContent implements ExecutablePolicyContent {

@JsonUnwrapped private final PolicyExecutionSpec executionSpec;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please do not introduce @JsonUnwrapped here. Gravitino currently has no precedent for this annotation, and it couples the public API domain model to a specific persistence JSON layout. Please follow the existing DTO/converter pattern: keep PolicyExecutionSpec Jackson-independent, flatten the execution fields explicitly in PolicyContentDTO or the persistence representation, and reconstruct the spec in DTOConverters or POConverters. The new Jackson dependency in the api module should then be removed as well.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks — addressed.

  • Removed @JsonUnwrapped and Jackson usage from api (PolicyExecutionSpec is Jackson-independent again).
  • Flat peer fields live on AbstractExecutablePolicyContent for the existing POConverters content JSON layout (same flat keys as the DTOs); executionSpec() reconstructs the in-memory spec.
  • Dropped the jackson.annotations dependency from the api module.

/** Represents a typed iceberg compaction policy content DTO. */
@EqualsAndHashCode
@ToString
@ToString(exclude = {"jobOptions", "updateOptions"})

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The redaction is incomplete: rewriteOptions is still rendered by both this Lombok-generated toString() and IcebergDataCompactionContent.toString(). Sensitive-key rejection is explicitly deferred, and the design now includes rewriteOptions among the bags that must reject credential-shaped keys, so an accidentally supplied credential can still be exposed in logs. Please exclude or redact rewriteOptions as well (keys/count only, no values).

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch — fixed.

  • IcebergDataCompactionContent.toString() now prints rewriteOptionsKeys=... only (no values), consistent with jobOptions / updateOptions.
  • DTO @ToString excludes rewriteOptions as well (same as the other option bags).

Sensitive-key rejection on create/alter remains the planned follow-up.

@JsonProperty("minIntervalMs") @Nullable Long minIntervalMs,
@JsonProperty("jobOptions") @Nullable Map<String, String> jobOptions,
@JsonProperty("updateOptions") @Nullable Map<String, String> updateOptions) {
if (!PolicyExecutionFields.hasAny(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hasAny treats false as unset, so PolicyExecutionSpec.of(null, false, ...) returns EMPTY and changes false to null. But if any other peer is present, the same false is retained. This makes JSON round trips and equals/hashCode behavior depend on unrelated fields. Please either normalize false to null unconditionally, or treat every non-null Boolean as an explicitly supplied value.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed — fixed by treating every non-null Boolean as an explicit value (including false).

  • PolicyExecutionFields.hasAny / PolicyExecutionSpec.of now retain lone false instead of collapsing to EMPTY/null.
  • Clarified DTO Javadoc: null = unset, false = explicitly disabled, true = enabled.

lasdf1234 and others added 2 commits October 15, 2026 16:35
…, fix false peers

Store flat execution peer fields on content for persistence without Jackson
in api; treat non-null commitTrigger (including false) as explicitly set;
omit rewriteOptions values from toString paths.

Co-authored-by: Cursor <cursoragent@cursor.com>
…TO Javadoc

Co-authored-by: Cursor <cursoragent@cursor.com>
@lasdf1234

Copy link
Copy Markdown
Collaborator Author

Thanks @roryqi for the follow-up review — latest push addresses the three new notes:

  1. No @JsonUnwrapped — PolicyExecutionSpec is Jackson-independent again; flat peers on content + DTO/converter reconstruction; removed jackson.annotations from api.
  2. rewriteOptions toString redaction — keys only on the API type; excluded from DTO @ToString (credential rejection still follow-up).
  3. commitTrigger=false consistency — non-null Boolean is always retained; Javadoc clarified (null unset / false explicit off).

Please take another look when you have a chance.

@Nullable private final Boolean commitTrigger;
@Nullable private final Long minIntervalMs;
private final Map<String, String> jobOptions;
private final Map<String, String> updateOptions;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updateOptions or updaterOptions?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Renamed to `updaterOptions` (JSON key + API) to align with runtime `updater_options` / `updaterOptions()`. Design doc updated in the same commit.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got. Rename to updaterOptions.

@roryqi roryqi left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please do not flatten PolicyExecutionSpec into the domain model and every concrete DTO just to match the persisted/wire JSON shape. Follow the existing JsonUtils.DistributionSerializer / DistributionDeserializer pattern instead: keep execution settings as an immutable composed value object, and use a custom serializer/deserializer in common to map its fields to/from the flat JSON keys (timeTrigger, commitTrigger, minIntervalMs, jobOptions, and updateOptions). That keeps Jackson and persistence-layout concerns out of api, avoids duplicating the same five fields/accessors across all executable policy DTOs, and preserves the current flat JSON contract. Please add REST DTO and POConverters round-trip coverage for each executable built-in policy type.

lasdf1234 and others added 2 commits October 15, 2026 19:12
…Options

Align the policy execution peer with runtime updater_options / updaterOptions
naming used by the update-stats path, and update the TMS design doc.

Co-authored-by: Cursor <cursoragent@cursor.com>
…ap flat JSON via SerDe

Avoid flattening execution peers across domain/DTO types. Persist and wire flat
keys through ExecutablePolicyContentSerde, and add REST/POConverters round-trips
for each executable built-in policy.

Co-authored-by: Cursor <cursoragent@cursor.com>
@lasdf1234

Copy link
Copy Markdown
Collaborator Author

Split for review / merge into `branch-tms-scheduler`:

  1. [#13800] feat(policy): Maintenance execution peers on Iceberg contents #13885 — execution peers (domain / DTO / SerDe)
  2. [#13800] feat(policy): BuiltInType minIntervalMs defaults registry #13886 — BuiltInType `minIntervalMs` defaults registry

Closing this PR in favor of the split. Original head `feature/tms-policy-peers-13800` is preserved on the fork.

@lasdf1234

Copy link
Copy Markdown
Collaborator Author

Superseded by split PRs #13885 and #13886 (base: branch-tms-scheduler).

@lasdf1234 lasdf1234 closed this Oct 15, 2026
roryqi pushed a commit that referenced this pull request Oct 16, 2026
#13885)

### What changes were proposed in this pull request?

Split from #13801 (1/2). **Domain** execution peers for Epic #13764 /
#13800.

- `PolicyExecutionSpec` / `ExecutablePolicyContent` /
`PolicyExecutionFields`
- Four Iceberg maintenance contents carry peers at construction
- TMS design notes on peers / `BuiltInType` identity

**Follow-up (stack):** #13888 / #13887 — wire/DTO/SerDe +
`minIntervalMs` registry.

### Why are the changes needed?

TMS needs shared trigger / cooldown / option-bag concepts without
freezing them onto every `PolicyContent`.

Fix: #13800

### Does this PR introduce _any_ user-facing change?

API/domain surface for optional peers on Iceberg maintenance contents
(REST wire lands in #13887).

### How was this patch tested?

```bash
./gradlew :api:test --tests 'org.apache.gravitino.policy.*' -PskipITs
```

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
roryqi pushed a commit that referenced this pull request Oct 16, 2026
…s defaults (#13887)

### What changes were proposed in this pull request?

Split from #13801 (2/2). **Stack on #13885** (merge that first). Tracked
by #13888.

Incremental scope (~1.5k vs #13885 ~1.1k):

- Flat JSON mapping via `ExecutablePolicyContentSerde` + Iceberg
`PolicyContentDTO` / `DTOConverters`
- REST + `POConverters` round-trips for each executable built-in type
- `MaintenancePolicyDefinitions` + per-type
`gravitino.maintenance.*.minIntervalMs` + conf/docs

Until #13885 merges, the GitHub file list includes domain files from the
stack base; review the **second commit** /
`common`+`core/maintenance`+`conf`+`docs` paths.

### Why are the changes needed?

Preserve flat wire/persistence JSON without flattening Spec into every
type; server defaults for omitted `minIntervalMs`.

Fix: #13888

### Does this PR introduce _any_ user-facing change?

Yes — optional JSON peer fields on Iceberg maintenance policy contents,
and:

- `gravitino.maintenance.compaction.minIntervalMs` (`3600000`)
- `gravitino.maintenance.manifest-rewrite.minIntervalMs` (`3600000`)
- `gravitino.maintenance.snapshot-expiry.minIntervalMs` (`3600000`)
- `gravitino.maintenance.orphan-cleanup.minIntervalMs` (`86400000`)

### How was this patch tested?

```bash
./gradlew :common:test --tests 'org.apache.gravitino.dto.policy.*' \
  :core:test --tests 'org.apache.gravitino.storage.relational.utils.TestPOConverters' \
  --tests 'org.apache.gravitino.maintenance.policy.TestMaintenanceMinIntervalConfigs' \
  -PskipITs
```

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Subtask] TMS: Domain PolicyExecutionSpec and executable peers

2 participants