You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit abac270
Browse filesBrowse the repository at this point in the historyBrowse files
[#11196] feat(maintenance): Collect manifest statistics in single-writer table cycles (#13717)
### What changes were proposed in this pull request?
Rebase the collector directly onto current main and remove the
dependency on #13532. Collect every spec from one pinned table snapshot
with one `allManifests` traversal. Publish both complete maps and
`custom-manifest-snapshot-id` in one existing PUT/update call.
Default to last-write-wins publication without a coordination directory.
Consumers requiring fresh statistics must compare the published snapshot
ID with the table's current snapshot and skip/recollect on mismatch. An
optional shared coordination directory retains per-table locking through
publication and explicit recovery after ambiguous writes. Collection
failures are always rerunnable. The collector forces
`gravitino.client.retryTimes=0`; other clients retain the default of one
retry.
### Why are the changes needed?
Follow the single-writer direction agreed in #13532. Avoid scanning the
same manifest list once per spec and introducing a public merge API
solely for this collector. Related to #11196 and design #12942.
### Does this PR introduce _any_ user-facing change?
Adds `builtin-iceberg-update-manifest-stats` with optional
`coordination_directory`. Omitted or blank uses last-write-wins; opting
in requires a shared filesystem supporting durable files and cross-host
Java/POSIX locks. Collection is table-wide with no `spec_id` option.
Defined specs without manifests publish zeroes; each update replaces the
complete maps, removing stale entries. Snapshot ID is -1 when no
snapshot exists. Automatic policy triggering remains separate.
### How was this patch tested?
- 222 jobs tests, 19 optimizer API tests, and 30 focused Java client
tests passed.
- Coverage includes partition evolution, empty tables, one
snapshot/manifest pass, independent processes contending for the lock,
uncoordinated publication and reruns, safe reruns after collection
failures, ambiguous PUT recovery, auth alias propagation, and HTTP retry
counts of zero, one, and two.
- Formatting, jobs Javadocs, `:docs:build`, `rat`, and `git diff
--check` passed.
- Cross-host filesystem failure behavior was not exercised; deployments
must verify shared lock/durability semantics.
|`gravitino.client.connectionTimeoutMs`| An optional http connection timeout in milliseconds. |`180000`(3 minutes) | No |
45
+
|`gravitino.client.retryTimes`| Maximum automatic HTTP retries after the initial request. Must be a nonnegative integer; `0` disables retries. |`1`| No |
45
46
|`gravitino.client.socketTimeoutMs`| An optional http socket timeout in milliseconds. |`180000`(3 minutes) | No |
46
47
47
48
**Note:** Invalid configuration properties will result in exceptions.
Five job templates ship with the service, and they are complementary rather than alternatives. A full maintenance pass collects statistics, compacts data files, expires the snapshot history that compaction just created, consolidates manifests, and removes old orphan files.
240
+
The following Iceberg job templates ship with the service. A full maintenance pass collects statistics, compacts data files, expires the snapshot history that compaction just created, consolidates manifests, and removes old orphan files.
Each can be submitted directly over REST, and the first two are also what the policy-driven workflow submits on your behalf. See [Quick Start](./optimizer.md#walkthrough) for the policy-driven path.
251
252
@@ -282,6 +283,111 @@ table, so it must be asked for explicitly.
282
283
283
284
Its `jobConf` is documented in [Configuration](./optimizer-configuration.md#job-submission-configuration).
284
285
286
+
## Update Manifest Statistics
287
+
288
+
`builtin-iceberg-update-manifest-stats` is a separate Spark job that collects two table-level
289
+
statistics, including for partitioned tables:
290
+
291
+
| Statistic | Object entry for each decimal spec ID |
292
+
| --- | --- |
293
+
|`custom-manifest-number-by-spec`| Long manifest count |
294
+
|`custom-avg-manifest-size-by-spec`| Double average manifest size in bytes |
295
+
296
+
The job reads manifest metadata without scanning file statistics. It works after partition
297
+
evolution, including for specs whose partition fields are absent from older files.
298
+
`builtin-iceberg-update-stats` and the `submit-update-stats-job` CLI retain their existing behavior;
299
+
they do not collect manifest statistics. Submit the new template directly through the jobs API.
300
+
301
+
Each table cycle pins one current snapshot, calls `allManifests` once, and groups data and
302
+
delete manifests by `partitionSpecId`. Every defined spec is included. A spec with no manifests,
303
+
including a table without a snapshot, produces count `0` and average `0.0`.
304
+
The job also publishes `custom-manifest-snapshot-id` in the same update (`-1` without a snapshot).
305
+
Snapshot IDs identify measurements; they are not monotonically increasing version numbers.
306
+
307
+
The template accepts `catalog_name`, `table_identifier`, `updater_options`, and optional
308
+
`coordination_directory` and `spark_conf`, alongside the standard Spark template settings. There is no `spec_id` filter:
309
+
collection always replaces the complete maps for the table.
310
+
`updater_options` is a JSON string containing `gravitino_uri` and `metalake`; it can also select
311
+
a `statistics_updater` implementing the existing table statistics update operation.
312
+
For example:
313
+
314
+
```bash
315
+
curl -X POST http://localhost:8090/api/metalakes/test/jobs/runs \
For example, one collection can produce complete maps `{"0": 620, "1": 120}` and
336
+
`{"0": 10485760.0, "1": 4194304.0}`. The collector owns both statistic names and publishes
337
+
them together with the snapshot ID in one existing PUT/update call. Old map entries are
338
+
replaced, not merged. Other statistic names are unchanged. No PATCH or merge API is needed.
339
+
340
+
By default, `coordination_directory` is omitted or blank and publications use last-write-wins
341
+
semantics. Each update contains the complete maps and snapshot ID; a concurrent older collection
342
+
may overwrite newer measurements. Failed jobs can simply be rerun, with no lock files or recovery.
343
+
Consumers requiring fresh statistics must compare `custom-manifest-snapshot-id` with the table's
344
+
current snapshot ID (`-1` if none) and skip/recollect on a mismatch. This is a consumer requirement,
345
+
not a freshness check enforced by this collector. Snapshot IDs must be compared for equality,
346
+
not numerical ordering.
347
+
348
+
### Optional shared coordination and recovery
349
+
350
+
For stricter serialization, provision `coordination_directory` (`--coordination-dir` for direct Spark invocation) as an
351
+
existing absolute directory on a shared filesystem that supports durable files and cross-host
352
+
Java/POSIX file locks. Every runner for a destination table must use the same shared directory,
353
+
metalake, and canonical catalog/schema/table names. A host-local directory, object-store mount,
354
+
or filesystem without reliable cross-host locking does not satisfy this requirement.
355
+
All writers of these statistic names must opt into this mode and use the same directory;
356
+
mixing coordinated and uncoordinated writers does not provide serialization.
357
+
358
+
The job locks a persistent file keyed by metalake and table before refreshing the Iceberg table
359
+
or choosing a snapshot, and holds it through the PUT. Concurrent runs for that table fail;
360
+
other tables can proceed. Snapshot IDs are recorded for provenance, not compared numerically.
361
+
The job forces `gravitino.client.retryTimes=0` for its Gravitino client.
362
+
Custom statistics updaters and any intervening proxies must likewise avoid retrying PUT requests;
363
+
an automatic resend could hide an original request that is still executing. There are no expiring
364
+
leases, automatic retries, scheduling, or cooldowns.
365
+
366
+
Collection failures can be rerun normally: the lock is released without leaving a recovery marker.
367
+
After collection and immediately before calling the updater, the job durably writes an in-progress
368
+
marker. It clears the marker only after a successful update. A crash after the marker is written or
369
+
an update failure leaves the marker in place, even after the operating system releases the lock.
370
+
The updater interface cannot distinguish a failed request from a request still executing on the
371
+
server, so these failures require recovery. The error reports the recovery file path.
372
+
373
+
For example, run A sends a PUT and loses its connection while the server is still processing it.
374
+
An immediate rerun B could publish newer measurements before A's delayed PUT overwrites them.
375
+
The file lock prevents concurrent collection across hosts; the marker prevents this overlap after
376
+
an ambiguous failure. Snapshot IDs provide provenance, but the existing PUT does not enforce
377
+
snapshot ordering or fencing. Automatic reruns after ambiguous writes would require server-side
378
+
conditional publication or an equivalent guarantee that the earlier request has ended.
379
+
380
+
For recovery, stop all runners for that table and verify the previous process has terminated
381
+
and any outstanding Gravitino PUT has completed or been definitively cancelled. While holding
382
+
an exclusive lock on the reported file, truncate it to zero bytes and flush it to durable storage,
383
+
then resume collection. Never delete or replace the file: a new inode can permit two independent
384
+
locks. If the previous PUT outcome cannot be established, leave the marker intact.
385
+
386
+
Consumers must read both objects from one table-statistics response and use the same chosen
387
+
spec key in each. A missing object or key means collection is required, not zero manifests.
388
+
`IcebergManifestStatistics.fromStatistics` returns an empty result for an incomplete pair.
389
+
Automatic manifest policy triggering is separate follow-up work.
390
+
285
391
## Rewrite Data Files
286
392
287
393
`builtin-iceberg-rewrite-data-files` performs the compaction itself, merging small data files into larger ones. It is what a compaction policy submits when its thresholds are crossed.
0 commit comments