Core: copy ByteBuffers when copying GenericRecord (cherry-pick apache/iceberg#12855)

[abrarsher23]

What

Straight git cherry-pick -x of apache/iceberg a7f3dc79a2f42a4875ac35eec2137ecff15204fc
("Core: Copy ByteBuffers when Copying GenericRecord", PR apache#12855), the fix for
apache/iceberg#11239. One production file,
+16/-1; the other three files are the upstream regression tests that ship with the commit.
No conflicts, no adaptation.

The follow-on commits are release plumbing only: version.txt (1.8.1-PATCH.4), an AFFIRM_VERSIONS.md for the
1.8.1 line (backfilled with PATCH.1 / PATCH.3) following the convention on 1.11.0-affirm-patch, and the matching
dev/.rat-excludes entry. After merge I will tag 1.8.1-PATCH.4 at the merge sha (the first tag on this line).

Why

BaseDeleteLoader opens equality-delete files with reuseContainers()
(data/src/main/java/org/apache/iceberg/data/BaseDeleteLoader.java:245) and relies on
Record::copy at :131 to detach each record before building the StructLikeSet at :114.
At 1.8.1 that copy is shallow (Arrays.copyOf over the value array), so for a binary column
every copied record still points at the single ByteBuffer that ParquetValueReaders$BytesReader
reuses — and which, by the end of the file, holds the last record's bytes. The set collapses to
one effective key per delete file and every other delete in that file is silently dropped.

Two consequences for tables whose equality key includes a binary column:

• rows that an equality delete should hide are returned (duplicate keys on read), and a compaction running on
  the same reader makes those duplicates permanent;
• on a composite key, each record's non-binary components are paired with the file's last binary value,
  producing keys no delete targeted, so live rows can be hidden. This is a silent data-loss fix, not only a
  dedup fix.

Verification

Unit tests (local, JDK 17, Gradle 8.12.1)

• ./gradlew :iceberg-data:test — 602 tests, 0 failures, 0 errors, 58 skipped.
• TestGenericRecord: 4/4 incl. the new testDeepCopyValues().
  TestGenericReaderDeletes: 44 tests, 0 failures, 1 pre-existing skip.
• Negative control: reverting only GenericRecord.java while keeping the new tests reproduces the
  bug — 8 failures in TestGenericReaderDeletes (binary + struct equality-delete cases, all three file
  formats) and testDeepCopyValues fails with Expected not same: java.nio.HeapByteBuffer[pos=0 lim=12 cap=12].

Integration test (real tables, same snapshots, before vs after)

• Built the Spark runtime jar from this branch and ran it through our normal Spark build and a test deployment.
  The running JVM reports Apache Iceberg 1.8.1-PATCH.4 at this branch's head commit.
• Re-read tables that had shown the problem, pinned to the same snapshots that had previously been read with
  1.8.1-PATCH.3, so the data and delete files were identical and only the reader changed.
• With 1.8.1-PATCH.3 those reads returned duplicate keys; with this patch the duplicates that the table's
  equality deletes should suppress are gone. The results match an independent merge-on-read computed from the
  same files, on every table checked.
• Duplicates that are physically present with no covering delete (already made permanent by an earlier
  compaction) are unchanged, as expected: a reader fix does not repair history.

Notes for the reviewer

• Pre-existing red CI, unrelated to this change:
  • :iceberg-core:spotlessJavaCheck already fails on 1.8.1-affirm-patch for
    ExponentialHttpRequestRetryStrategy.java, TestExponentialHttpRequestRetryStrategy.java and
    TestRESTCatalog.java (from Prevent Retries 5xx Errors for Non-Idempotent Requests #1). Identical at the base commit; untouched here; deliberately no spotlessApply.
  • core-tests / kafka-connect-tests fail on pull access denied for minio/minio (Docker Hub), same as on [LAKE-6732] Spark 3.5: refuse to rewrite a table with duplicate data file registrations #6.
    The 1.11 line fixed this with Build: Pull MinIO images from Quay (cherry-pick apache/iceberg#18071) #11 (Quay images); a separate port to this line would green it.
• Cost: a real deep copy per delete record — one small ByteBuffer allocation per equality-delete row loaded
  for binary columns. Not benchmarked; worth knowing for a very wide binary-heavy table.
• Residual upstream defect (unchanged by Core: Deep copy Record values for equality deletes apache/iceberg#12855): the
  GenericRecord(GenericRecord, Map<String,Object> overwrite) overload is still Arrays.copyOf.
  BaseDeleteLoader uses the no-arg Record::copy, so the delete path is fixed; callers of
  copy(overwriteValues) on a binary field are not. Same on 1.11.0-affirm-patch.
• 1.11.0-affirm-patch already contains this fix (1.10.0+), so this commit disappears on a rebase to that base.
  It is deliberately a stopgap for the 1.8.1 line.

---

Change Management

• CMT approvals happen on this PR — single CMT for the whole change; the downstream version-bump PR references it.

What changes: the Spark reader/compaction path picks up 1.8.1-PATCH.4 = 1.8.1-PATCH.3 + this one commit.
No API, file-format or table-metadata change.

Risk: Low–Medium. One upstream commit with its own regression tests, validated before/after on real tables.
The behavioural change is confined to GenericRecord.copy() for ByteBuffer and nested-record values; all other
values keep identical references. Main risk is the manual jar → Spark package release path, mitigated by a dev-build
dry run that was deployed and exercised before this PR.

Rollback: re-pin the downstream package to the previous build (1.8.1-PATCH.3). Nothing is persisted in a
new format, so a rolled-back reader simply resumes the old behaviour.

Testing: unit tests + negative control + before/after integration read (above).

Deployment order: merge → tag 1.8.1-PATCH.4 → build + publish the release package → downstream version bump
(separate PR) → deploy.

[abrarsher23]

Superseded by #18 (same commits, renamed branch).