[Kernel] Fix schema validation for new nullable structs with non-nullable fields

[c-kaustubh]

Which Delta project/connector is this regarding?

• Spark
• Standalone
• Flink
• Kernel
• Other (fill in here)

Description

Schema evolution validation incorrectly rejects adding a new nullable struct
that contains non-nullable inner fields, throwing:
KernelException: "Cannot add non-nullable field <fieldname>"

This happens because computeSchemaChangesById uses post-order traversal via
SchemaIterable, and struct children are recorded independently in addedFields
(each with their own column mapping ID via getNearestStructFieldAncestor()
returning self for StructSchemaZipper). The validator then sees the inner
non-nullable field and rejects it, even though the entire containing struct is
new and nullable -- no existing data could violate the constraint.

This already works correctly for arrays and maps because their synthetic children
(key/value/element) share the parent's field ID via getNearestStructFieldAncestor()
walking up to the parent struct field. Only struct children are affected.

Example that fails before this fix:

// Old schema: { a: int }
// New schema: { a: int, my_struct(nullable): { inner(NOT NULL): string } }
// Fails with: "Cannot add non-nullable field inner"
// Should succeed: my_struct is new and nullable, inner constrains future data only

Fix: In validateUpdatedSchemaCompatibility, before checking nullability of
added fields, build a descendantFieldIds set by walking each added field's
DataType tree via a new collectDescendantFieldIds method. Fields whose ID is in
this set have their nullability check skipped -- they are descendants of entirely
new containers and their nullability only constrains future writes. The fieldId <= maxFieldId constraint is still enforced unconditionally for all added fields.

Only SchemaUtils.java is modified (no changes to computeSchemaChangesById,
SchemaIterable, SchemaChanges, or TransactionMetadataFactory).

Closes #6428

How was this patch tested?

9 new unit tests in SchemaUtilsSuite.scala covering:

• New nullable struct with non-nullable inner field (the core bug)
• Depth > 2 non-nullable chain (recursive skipSet)
• New nullable array with non-nullable struct element fields
• New nullable map with non-nullable struct value fields
• 4-level deep nesting: struct -> map -> array -> struct
• New struct inside existing struct (addedFields vs updatedFields boundary)
• 3 guard rails: non-nullable parent still rejected, error names parent not child,
  fieldId constraint still enforced for descendants

3 new integration tests in DeltaTableSchemaEvolutionSuite.scala covering:

• 4-level nesting with complex nested ID metadata end-to-end
• New struct inside existing struct alongside unchanged sibling end-to-end
• Core bug end-to-end through TransactionBuilder (updated existing test)

1 existing integration test updated: "Adding non-nullable field fails" was
asserting the buggy behavior (rejecting a valid schema evolution). Updated to
assert success and verify the non-nullable property is preserved.

All tests pass across SchemaUtilsSuite (68) and DeltaTableSchemaEvolution* (98). No regressions.

Does this PR introduce any user-facing changes?

Yes. Schema evolution via Delta Kernel now correctly accepts adding a new nullable
struct (or array/map) that contains non-nullable inner fields. Previously this was
rejected with "Cannot add non-nullable field <name>".

This aligns with the behavior already supported for arrays and maps (where inner
struct fields with non-nullable constraints are accepted when the container is new),
and is consistent with Delta Spark's behavior.

No existing valid rejections are affected:

• Adding non-nullable fields to existing structs still fails
• Adding non-nullable fields to structs inside existing arrays/maps still fails
• Tightening nullability on existing fields still fails
• Type changes on existing fields still fail (unless type widening is enabled)
• REPLACE TABLE behavior is unchanged (allowNewRequiredFields=true bypasses the
  check before the skipSet is consulted)