Skip to content

[Spark] Concurrent Identity Columns: add client support for the UC Identity Sequence service - #7785

Draft
Rob2U wants to merge 16 commits into
delta-io:masterfrom
Rob2U:stack/cic-export-7-client
Draft

[Spark] Concurrent Identity Columns: add client support for the UC Identity Sequence service #7785
Rob2U wants to merge 16 commits into
delta-io:masterfrom
Rob2U:stack/cic-export-7-client

Conversation

@Rob2U

@Rob2U Rob2U commented Oct 1, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Which Delta project/connector is this regarding?

  • Spark
  • Standalone
  • Flink
  • Kernel
  • Other (fill in here)

Description

Adds a client for Unity Catalog identity-sequence support of Concurrent Identity Columns.

  • Extends UCDeltaClient with createIdentitySequence, reserveIdentityIds, dropIdentitySequence, and IdentitySequenceRange.
  • Implements the REST calls in UCDeltaTokenBasedRestClient using the Delta Identity Sequences API, with validation and error mapping.
  • Introduces UCDeltaIdentitySequenceService, the OSS IdentitySequenceService adapter that delegates to UCDeltaClient via the UC commit-coordinator client factory.

This is the last PR / branch of the concurrent identity columns feature (see #7618). It also relies on the unity catalog integration of the service (see unitycatalog/unitycatalog#1924).
The combined stack adds support for the Concurrent Identity Column (CIC) feature following the RFC here.
The TLDR for this RFC/feature: It allows for two concurrent 'inserters' to both succeed by adding an identity sequence service that hands out disjoint ranges of id values. This way the two writers do not conflict and can both commit.

Stack overview:

  1. Foundation
    ([Spark] Foundation for Concurrent Identity Columns: metadata key, error classes #7618)
  2. Service abstraction + In-Memory one
    ([Spark] Concurrent Identity Columns: sequence-service abstraction + in-memory backend #7619)
  3. Transport and buffering for the service requests
    ([Spark] Concurrent Identity Columns: executors can reserve ranges from the driver (buffer) #7620)
  4. Table feature and the insert path
    ([Spark] Concurrent Identity Columns: INSERT/append write path and table feature #7621)
  5. DDL integration
    ([Spark] Concurrent Identity Columns: DDL integration (CREATE/REPLACE/CONVERT/SYNC/DROP COLUMN) #7622)
  6. Merge support <- this PR
    ([Spark] Concurrent Identity Columns: add support for Merge #7623)
  7. UC Client addition
    ([Spark] Concurrent Identity Columns: add client support for the UC Identity Sequence service  #7785)

How was this patch tested?

Unit tests for the REST client identity-sequence methods were added.

Does this PR introduce any user-facing changes?

No

@Rob2U Rob2U changed the title [WIP][CIC-7] add client support for the UC Identity Sequence service [Spark] Concurrent Identity Columns: add client support for the UC Identity Sequence service Oct 1, 2026
This was referenced Oct 1, 2026
Merged
Merged
Open
Open
Open
Rob2U and others added 7 commits October 2, 2026 08:19
@Rob2U @mkroll-db
[CIC] Foundation: metadata key, error classes, config surface
a13ccb8 
Co-authored-by: mkroll-db <marco.kroll@databricks.com>
Signed-off-by: Robert Weeke <weeke.robert@gmail.com>
@Rob2U
[CIC] Foundation: refresh error classes
9732bf6 
Signed-off-by: Robert Weeke <weeke.robert@gmail.com>
@Rob2U
[CIC] Foundation: add sequenceId vs. highWaterMark conflict detection…
ba19ae3 
… and resolution

Signed-off-by: Robert Weeke <weeke.robert@gmail.com>
@Rob2U @mkroll-db
[CIC] Backend seam: IdentitySequenceService trait, fake, resolve factory
27ef904 
Co-authored-by: mkroll-db <marco.kroll@databricks.com>
Signed-off-by: Robert Weeke <weeke.robert@gmail.com>
@Rob2U
[CIC] Backend seam: add a test helper used by the e2e suites.
8ae9038 
Signed-off-by: Robert Weeke <weeke.robert@gmail.com>
@Rob2U @mkroll-db
[CIC] Transport: coordinator, client, reserve buffer, observability
41d1e13 
Co-authored-by: mkroll-db <marco.kroll@databricks.com>
Signed-off-by: Robert Weeke <weeke.robert@gmail.com>
@Rob2U @mkroll-db
[CIC] Write path: reservation, generation, default-expr integration, …
41ffa03 
…TransactionalWrite

Co-authored-by: mkroll-db <marco.kroll@databricks.com>
Signed-off-by: Robert Weeke <weeke.robert@gmail.com>
@Rob2U
Rob2U force-pushed the stack/cic-export-7-client branch from 520a920 to 5a2d354 Compare October 4, 2026 21:52
Rob2U and others added 9 commits October 5, 2026 10:50
@Rob2U
[CIC] Write path: skip reservation for tables without the feature or …
b6208fe 
…sequence pointers

Signed-off-by: Robert Weeke <weeke.robert@gmail.com>
@Rob2U @mkroll-db
[CIC] DDL: create-table hook, conversion, SYNC IDENTITY, command inte…
548e7ad 
…gration + conversion suite

Co-authored-by: mkroll-db <marco.kroll@databricks.com>
Signed-off-by: Robert Weeke <weeke.robert@gmail.com>
@Rob2U
[CIC] DDL: add create-table hook / conversion / SYNC refinements and …
2dd54f5 
…expande conversion-suite coverage

Signed-off-by: Robert Weeke <weeke.robert@gmail.com>
@Rob2U
[CIC] DDL: update with import fix, suite catalog wiring, edge-gated D…
bf07383 
…ROP FEATURE test

Signed-off-by: Robert Weeke <weeke.robert@gmail.com>
@Rob2U
[CIC] DDL: SYNC IDENTITY repairs a column whose feature was removed b…
6c3a86f 
…ut kept its sequence pointer

Signed-off-by: Robert Weeke <weeke.robert@gmail.com>
@Rob2U @mkroll-db
[CIC] MERGE: reservation hook + e2e ServiceBackend and MergeInto suites
4ecd4ff 
Co-authored-by: mkroll-db <marco.kroll@databricks.com>
Signed-off-by: Robert Weeke <weeke.robert@gmail.com>
@Rob2U
[CIC] MERGE: change ServiceBackend/MergeInto e2e suites to catalog-ma…
be495aa 
…naged (CatalogOwned) tables and use the real metadata.id

Signed-off-by: Robert Weeke <weeke.robert@gmail.com>
@Rob2U
[CIC] MERGE: detect orphaned sequence pointer before wrong-generator …
2e5bab3 
…error and add service-backed reservation test

Signed-off-by: Robert Weeke <weeke.robert@gmail.com>
@Rob2U
[CIC] UC backend: UCDeltaClient identity-sequence operations and OSS …
7612503 
…service adapter

Adds Unity Catalog-backed identity-sequence support for Concurrent Identity Columns:
- Extends `UCDeltaClient` with `createIdentitySequence`, `reserveIdentityIds`, `dropIdentitySequence`, and `IdentitySequenceRange`.
- Implements the REST calls in `UCDeltaTokenBasedRestClient` using the Delta Identity Sequences API, with validation and error mapping.
- Introduces `UCDeltaIdentitySequenceService`, the OSS `IdentitySequenceService` adapter that delegates to `UCDeltaClient` via the UC commit-coordinator client factory.
- Adds unit tests for the REST client identity-sequence methods.

Signed-off-by: Robert Weeke <weeke.robert@gmail.com>
@Rob2U
Rob2U force-pushed the stack/cic-export-7-client branch from 5a2d354 to 7612503 Compare October 5, 2026 14:08
@murali-db
murali-db requested a review from yili-db October 7, 2026 16:24
murali-db pushed a commit that referenced this pull request Oct 14, 2026
@Rob2U @mkroll-db
[Spark] Foundation for Concurrent Identity Columns: metadata key, err…
4e0d160 
…or classes (#7618)

<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->

#### Which Delta project/connector is this regarding?
<!--
Please add the component selected below to the beginning of the pull
request title
For example: [Spark] Title of my pull request
-->

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description
This PR is the first out of a stack of seven PRs in total. 
The combined stack adds support for the Concurrent Identity Column (CIC)
feature following the RFC
[here](#7272).
The TLDR for this RFC/feature: It allows for two concurrent 'inserters'
to both succeed by adding an identity sequence service that hands out
disjoint ranges of id values. This way the two writers do not conflict
and can both commit.

**_Stack overview:_** 
1. Foundation <- this PR
    (#7618)
2. Service abstraction + In-Memory one
    (#7619)
3. Transport and buffering for the service requests 
    (#7620)
4. Table feature and the insert path
    (#7621)
5. DDL integration 
    (#7622)
6. Merge support
    (#7623)
7. UC Client addition
    (#7785) 


This PR introduces no behaviour changes, just scaffolding. It adds:
- `ConcurrentIdentityColumnSchema`: the
`delta.identity.concurrent.sequenceId` column-metadata key that marks a
column as a CIC.
- `ConcurrentIdentityColumnErrors` + 8
DELTA_CONCURRENT_IDENTITY_COLUMN_* error classes (disabled,
too-many-columns, sequence-not-found, metadata-mismatch,
empty-reserve-range, service-timeout, conversion-incomplete,
using-wrong-generator).
- Config
`spark.databricks.delta.identityColumn.concurrent.maxColumnsPerTable`

## How was this patch tested?

<!--
If tests were added, say they were added here. Please make sure to test
the changes thoroughly including negative and positive cases if
possible.
If the changes were tested in any way other than unit tests, please
clarify how you tested step by step (ideally copy and paste-able, so
that other reviewers can test and check, and descendants can verify in
the future).
If the changes were not tested, please explain why.
-->
The ConcurrentIdentityColumnErrorsSuite adds three tests for the
`EMPTY_RESERVE_RANGE`, `METADATA_MISMATCH`, and `USING_WRONG_GENERATOR`
errors. The tests are only added for these errors because these errors
can not occur 'naturally' in the code and are therefore tested this way.


## Does this PR introduce _any_ user-facing changes?
<!--
If yes, please clarify the previous behavior and the change this PR
proposes - provide the console output, description and/or an example to
show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change
compared to the released Delta Lake versions or within the unreleased
branches such as master.
If no, write 'No'.
-->
No

---------

Signed-off-by: Robert Weeke <weeke.robert@gmail.com>
Co-authored-by: mkroll-db <marco.kroll@databricks.com>
murali-db pushed a commit that referenced this pull request Oct 15, 2026
@Rob2U @mkroll-db
[Spark] Concurrent Identity Columns: sequence-service abstraction + i…
df901ec 
…n-memory backend (#7619)

<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->

#### Which Delta project/connector is this regarding?
<!--
Please add the component selected below to the beginning of the pull
request title
For example: [Spark] Title of my pull request
-->

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description
This PR is the second out of a stack of seven PRs in total. The combined
stack adds support for the Concurrent Identity Column (CIC) feature
following the RFC [here](#7272).
TLDR: It allows for two concurrent 'inserters' to both succeed by adding
an identity sequence service that hands out disjoint ranges of id
values. This way the two writers do not conflict and can both commit.

**_Stack overview:_** 
1. Foundation
    (#7618)
2. Service abstraction + In-Memory one <- this PR
    (#7619)
3. Transport and buffering for the service requests 
    (#7620)
4. Table feature and the insert path
    (#7621)
5. DDL integration 
    (#7622)
6. Merge support
    (#7623)
7. UC Client addition
    (#7785) 



This PR introduces no behaviour changes, just scaffolding. It adds:
- IdentitySequenceService trait containing the methods `createSequence`,
`dropSequence`, `reserve` (+ CreateSequenceRequest / DropSequenceRequest
/ ReserveIdsRequest / ReserveIdsResponse).
- `IdentitySequenceServices.resolve(spark)` for retrieval of the service
to be used. Defaults to an in-memory `LocalIdentitySequenceService`,
overridable via
    `spark.databricks.delta.identityColumn.concurrent.serviceClassName`.

## How was this patch tested?
The `LocalIdentitySequenceService` is tested using the
`LocalIdentitySequenceServiceSuite.scala`. These tests ensure that the
the `LocalIdentitySequenceService` matches the expected behavior
exactly. This is important because we will use it later for testing the
other components.

<!--
If tests were added, say they were added here. Please make sure to test
the changes thoroughly including negative and positive cases if
possible.
If the changes were tested in any way other than unit tests, please
clarify how you tested step by step (ideally copy and paste-able, so
that other reviewers can test and check, and descendants can verify in
the future).
If the changes were not tested, please explain why.
-->


## Does this PR introduce _any_ user-facing changes?
<!--
If yes, please clarify the previous behavior and the change this PR
proposes - provide the console output, description and/or an example to
show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change
compared to the released Delta Lake versions or within the unreleased
branches such as master.
If no, write 'No'.
-->
No

---------

Signed-off-by: Robert Weeke <weeke.robert@gmail.com>
Co-authored-by: mkroll-db <marco.kroll@databricks.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@yili-db yili-db Awaiting requested review from yili-db

At least 1 approving review is required to merge this pull request.

Assignees

@mkroll-db mkroll-db

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@Rob2U @mkroll-db