Repository navigation
[#13800] feat(policy): Maintenance execution peers, BuiltInType registry, and minIntervalMs defaults - #13801
[#13800] feat(policy): Maintenance execution peers, BuiltInType registry, and minIntervalMs defaults#13801lasdf1234 wants to merge 20 commits into
Conversation
9c48d64 to
33f296b
Compare
1c6985e to
f7e2976
Compare
f7e2976 to
ea6e1ad
Compare
7f3a26c to
04e8141
Compare
04e8141 to
e8f52ec
Compare
…cy content Add TableMaintenanceContent with schedule, minIntervalMs, jobOptions, and updateOptions; wire the four Iceberg maintenance contents and PolicyContents factories; validate via TableMaintenancePolicies on PolicyEntity. Co-authored-by: Cursor <cursoragent@cursor.com>
e8f52ec to
851b289
Compare
api javadoc cannot resolve core's PolicyEntity; use {@code} instead so
:api:javadoc passes in CI.
Co-authored-by: Cursor <cursoragent@cursor.com>
…hedule triggers Move timeTrigger/eventTrigger/minIntervalMs/jobOptions/updateOptions onto PolicyContent as shared concepts, and drop the combined schedule object. Co-authored-by: Cursor <cursoragent@cursor.com>
Keep shared TimeTrigger/EventTrigger/PolicyExecutionFields on PolicyContent; remove TableMaintenanceContent, TaskType, and Policies without wiring peers onto Iceberg contents yet. Co-authored-by: Cursor <cursoragent@cursor.com>
…erver defaults Register gravitino.maintenance.task.*.minIntervalMs ConfigEntries with 1h/1d defaults so cooldown can be read from gravitino.conf without TaskType. Co-authored-by: Cursor <cursoragent@cursor.com>
| } | ||
|
|
||
| public Map<String, String> jobOptions() { | ||
| return jobOptions == null ? Collections.emptyMap() : jobOptions; |
There was a problem hiding this comment.
jobOptions() returns the DTO's mutable backing map, unlike rewriteOptions() above, which returns an unmodifiable defensive copy. A caller can therefore mutate the DTO after validation and also change its Lombok-generated equality/hash state. Please return an immutable defensive copy here and for updateOptions() in all four DTOs. Centralizing these fields in an immutable execution-spec DTO would also remove this repetition.
There was a problem hiding this comment.
Agreed — all four Iceberg execution DTOs now return PolicyExecutionFields.copyOptions(...) for jobOptions() / updateOptions() (unmodifiable defensive copies, same idea as rewriteOptions()). Added a regression test that mutating the returned maps throws.
| } | ||
|
|
||
| @Nullable | ||
| public String timeTrigger() { |
There was a problem hiding this comment.
These five new public accessors have no Javadocs, and the same pattern is repeated in the other execution-capable DTOs. Please document the new public API methods (including null/default semantics), or centralize them behind the proposed execution-spec abstraction.
There was a problem hiding this comment.
Agreed — centralized docs on PolicyContentDTO.ExecutionPeersDTO (null/empty semantics for the five peers). The four Iceberg DTOs implement that interface so the public accessors share one Javadoc contract.
roryqi
left a comment
There was a problem hiding this comment.
Additional review notes:
- The design document still defines triggers under
policy_version_info.content.scheduleand showsschedule.onCommit/schedule.crontab, while this implementation's wire contract is flatcommitTrigger/timeTrigger. Please make the design document and the actual JSON contract consistent. - The same section says automated maintenance must have at least one trigger and that create/alter rejects credential-shaped option keys. Neither rule is enforced by this PR. If those are intentionally deferred, the document should clearly mark them as future behavior rather than describing them as current guarantees.
- Please also make the cron syntax/example consistent with the scheduler contract when that validation is wired in.
The main blocking concern is the mutable, unchecked withExecutionFields API called out inline. An immutable PolicyExecutionSpec composed into executable policy contents would address that cleanly.
…ecutable peers Replace mutable withExecutionFields with construction-time PolicyExecutionSpec on ExecutablePolicyContent. Option map values are omitted from toString/DTO ToString to avoid leaking credentials in logs. Co-authored-by: Cursor <cursoragent@cursor.com>
Return unmodifiable option-map copies, document peers via ExecutionPeersDTO, and cover mutation attempts in TestPolicyDTO. Co-authored-by: Cursor <cursoragent@cursor.com>
Use commitTrigger/timeTrigger instead of content.schedule.*; drop the required-trigger rule; keep credential-key rejection as design intent. Co-authored-by: Cursor <cursoragent@cursor.com>
|
Thanks for the additional review notes — status:
|
Keep flat commitTrigger/timeTrigger peers; adopt scheduled_*_tasks wording from main. Co-authored-by: Cursor <cursoragent@cursor.com>
| private final Map<String, String> jobOptions; | ||
| private final Map<String, String> updateOptions; | ||
|
|
||
| private PolicyExecutionSpec( |
There was a problem hiding this comment.
PolicyExecutionSpec is now serialized as the executionSpec field of every executable policy content, but this type has only a private constructor and no Jackson creator/default constructor. POConverters.fromPolicyPO() therefore cannot read the stored content back; the current :core:test and Maintenance IT failures both report Cannot construct instance of PolicyExecutionSpec. Please make the persisted form deserializable (for example with an appropriate @JsonCreator/property mapping or a custom persistence representation) and add a POConverters round-trip test for maintenance policy content.
There was a problem hiding this comment.
Fixed — went with @JsonCreator on PolicyExecutionSpec.of(...) plus @JsonUnwrapped on AbstractExecutablePolicyContent.executionSpec, so peers stay flat on the wire (timeTrigger / commitTrigger / …) and POConverters.fromPolicyPO() can rebuild them.
Also added TestPOConverters.testFromPolicyPOIcebergCompactionWithExecutionPeers to cover the persistence round-trip (and assert the JSON is not nested under executionSpec).
…izable for POConverters Add @jsoncreator on PolicyExecutionSpec.of and @JsonUnwrapped on the content field so flat peer keys round-trip through JsonUtils.anyFieldMapper. Co-authored-by: Cursor <cursoragent@cursor.com>
| */ | ||
| public abstract class AbstractExecutablePolicyContent implements ExecutablePolicyContent { | ||
|
|
||
| @JsonUnwrapped private final PolicyExecutionSpec executionSpec; |
There was a problem hiding this comment.
Please do not introduce @JsonUnwrapped here. Gravitino currently has no precedent for this annotation, and it couples the public API domain model to a specific persistence JSON layout. Please follow the existing DTO/converter pattern: keep PolicyExecutionSpec Jackson-independent, flatten the execution fields explicitly in PolicyContentDTO or the persistence representation, and reconstruct the spec in DTOConverters or POConverters. The new Jackson dependency in the api module should then be removed as well.
There was a problem hiding this comment.
Thanks — addressed.
- Removed
@JsonUnwrappedand Jackson usage fromapi(PolicyExecutionSpecis Jackson-independent again). - Flat peer fields live on
AbstractExecutablePolicyContentfor the existing POConverters content JSON layout (same flat keys as the DTOs);executionSpec()reconstructs the in-memory spec. - Dropped the
jackson.annotationsdependency from theapimodule.
| /** Represents a typed iceberg compaction policy content DTO. */ | ||
| @EqualsAndHashCode | ||
| @ToString | ||
| @ToString(exclude = {"jobOptions", "updateOptions"}) |
There was a problem hiding this comment.
The redaction is incomplete: rewriteOptions is still rendered by both this Lombok-generated toString() and IcebergDataCompactionContent.toString(). Sensitive-key rejection is explicitly deferred, and the design now includes rewriteOptions among the bags that must reject credential-shaped keys, so an accidentally supplied credential can still be exposed in logs. Please exclude or redact rewriteOptions as well (keys/count only, no values).
There was a problem hiding this comment.
Good catch — fixed.
IcebergDataCompactionContent.toString()now printsrewriteOptionsKeys=...only (no values), consistent withjobOptions/updateOptions.- DTO
@ToStringexcludesrewriteOptionsas well (same as the other option bags).
Sensitive-key rejection on create/alter remains the planned follow-up.
| @JsonProperty("minIntervalMs") @Nullable Long minIntervalMs, | ||
| @JsonProperty("jobOptions") @Nullable Map<String, String> jobOptions, | ||
| @JsonProperty("updateOptions") @Nullable Map<String, String> updateOptions) { | ||
| if (!PolicyExecutionFields.hasAny( |
There was a problem hiding this comment.
hasAny treats false as unset, so PolicyExecutionSpec.of(null, false, ...) returns EMPTY and changes false to null. But if any other peer is present, the same false is retained. This makes JSON round trips and equals/hashCode behavior depend on unrelated fields. Please either normalize false to null unconditionally, or treat every non-null Boolean as an explicitly supplied value.
There was a problem hiding this comment.
Agreed — fixed by treating every non-null Boolean as an explicit value (including false).
PolicyExecutionFields.hasAny/PolicyExecutionSpec.ofnow retain lonefalseinstead of collapsing toEMPTY/null.- Clarified DTO Javadoc:
null= unset,false= explicitly disabled,true= enabled.
…, fix false peers Store flat execution peer fields on content for persistence without Jackson in api; treat non-null commitTrigger (including false) as explicitly set; omit rewriteOptions values from toString paths. Co-authored-by: Cursor <cursoragent@cursor.com>
…TO Javadoc Co-authored-by: Cursor <cursoragent@cursor.com>
|
Thanks @roryqi for the follow-up review — latest push addresses the three new notes:
Please take another look when you have a chance. |
| @Nullable private final Boolean commitTrigger; | ||
| @Nullable private final Long minIntervalMs; | ||
| private final Map<String, String> jobOptions; | ||
| private final Map<String, String> updateOptions; |
There was a problem hiding this comment.
updateOptions or updaterOptions?
There was a problem hiding this comment.
Renamed to `updaterOptions` (JSON key + API) to align with runtime `updater_options` / `updaterOptions()`. Design doc updated in the same commit.
There was a problem hiding this comment.
Got. Rename to updaterOptions.
There was a problem hiding this comment.
Please do not flatten PolicyExecutionSpec into the domain model and every concrete DTO just to match the persisted/wire JSON shape. Follow the existing JsonUtils.DistributionSerializer / DistributionDeserializer pattern instead: keep execution settings as an immutable composed value object, and use a custom serializer/deserializer in common to map its fields to/from the flat JSON keys (timeTrigger, commitTrigger, minIntervalMs, jobOptions, and updateOptions). That keeps Jackson and persistence-layout concerns out of api, avoids duplicating the same five fields/accessors across all executable policy DTOs, and preserves the current flat JSON contract. Please add REST DTO and POConverters round-trip coverage for each executable built-in policy type.
…Options Align the policy execution peer with runtime updater_options / updaterOptions naming used by the update-stats path, and update the TMS design doc. Co-authored-by: Cursor <cursoragent@cursor.com>
…ap flat JSON via SerDe Avoid flattening execution peers across domain/DTO types. Persist and wire flat keys through ExecutablePolicyContentSerde, and add REST/POConverters round-trips for each executable built-in policy. Co-authored-by: Cursor <cursoragent@cursor.com>
|
Split for review / merge into `branch-tms-scheduler`:
Closing this PR in favor of the split. Original head `feature/tms-policy-peers-13800` is preserved on the fork. |
#13885) ### What changes were proposed in this pull request? Split from #13801 (1/2). **Domain** execution peers for Epic #13764 / #13800. - `PolicyExecutionSpec` / `ExecutablePolicyContent` / `PolicyExecutionFields` - Four Iceberg maintenance contents carry peers at construction - TMS design notes on peers / `BuiltInType` identity **Follow-up (stack):** #13888 / #13887 — wire/DTO/SerDe + `minIntervalMs` registry. ### Why are the changes needed? TMS needs shared trigger / cooldown / option-bag concepts without freezing them onto every `PolicyContent`. Fix: #13800 ### Does this PR introduce _any_ user-facing change? API/domain surface for optional peers on Iceberg maintenance contents (REST wire lands in #13887). ### How was this patch tested? ```bash ./gradlew :api:test --tests 'org.apache.gravitino.policy.*' -PskipITs ``` --------- Co-authored-by: Cursor <cursoragent@cursor.com>
…s defaults (#13887) ### What changes were proposed in this pull request? Split from #13801 (2/2). **Stack on #13885** (merge that first). Tracked by #13888. Incremental scope (~1.5k vs #13885 ~1.1k): - Flat JSON mapping via `ExecutablePolicyContentSerde` + Iceberg `PolicyContentDTO` / `DTOConverters` - REST + `POConverters` round-trips for each executable built-in type - `MaintenancePolicyDefinitions` + per-type `gravitino.maintenance.*.minIntervalMs` + conf/docs Until #13885 merges, the GitHub file list includes domain files from the stack base; review the **second commit** / `common`+`core/maintenance`+`conf`+`docs` paths. ### Why are the changes needed? Preserve flat wire/persistence JSON without flattening Spec into every type; server defaults for omitted `minIntervalMs`. Fix: #13888 ### Does this PR introduce _any_ user-facing change? Yes — optional JSON peer fields on Iceberg maintenance policy contents, and: - `gravitino.maintenance.compaction.minIntervalMs` (`3600000`) - `gravitino.maintenance.manifest-rewrite.minIntervalMs` (`3600000`) - `gravitino.maintenance.snapshot-expiry.minIntervalMs` (`3600000`) - `gravitino.maintenance.orphan-cleanup.minIntervalMs` (`86400000`) ### How was this patch tested? ```bash ./gradlew :common:test --tests 'org.apache.gravitino.dto.policy.*' \ :core:test --tests 'org.apache.gravitino.storage.relational.utils.TestPOConverters' \ --tests 'org.apache.gravitino.maintenance.policy.TestMaintenanceMinIntervalConfigs' \ -PskipITs ``` --------- Co-authored-by: Cursor <cursoragent@cursor.com>
What changes were proposed in this pull request?
Domain + DTO slice of Epic #13764 / #13800.
AbstractExecutablePolicyContent(not universalPolicyContent):timeTrigger— cron string (syntax not validated inapi)commitTrigger— booleanminIntervalMs,jobOptions,updaterOptionsPolicyExecutionSpecAbstractExecutablePolicyContent.validate(); universalPolicyContent.validate()only checkssupportedObjectTypescommitTrigger == trueMaintenancePolicyDefinitionsregistry keyed byPolicy.BuiltInTypefor per-typeminIntervalMsserver defaults;resolve()rejects negative policy valuesPolicyContentDTO+DTOConvertersround-trip peers so REST alter does not drop themBuiltInTypeidentity vs scheduler task kinds (policy-expand/cron-job/commit-job)Out of scope (follow-ups):
Why are the changes needed?
TMS needs shared trigger / cooldown / option-bag concepts without freezing them onto every
PolicyContent. Server defaults cover omittedminIntervalMs.BuiltInTypestays the single maintenance identity (no parallel task-type enum).Fix: #13800
Does this PR introduce any user-facing change?
Yes.
New optional server config keys (with defaults):
gravitino.maintenance.compaction.minIntervalMs(3600000)gravitino.maintenance.manifest-rewrite.minIntervalMs(3600000)gravitino.maintenance.snapshot-expiry.minIntervalMs(3600000)gravitino.maintenance.orphan-cleanup.minIntervalMs(86400000)Optional JSON peer fields on Iceberg maintenance policy contents (REST/DTO):
{ "timeTrigger": "0 0 2 * * ?", "commitTrigger": true, "minIntervalMs": 3600000, "jobOptions": {}, "updaterOptions": {} }How was this patch tested?