Repository navigation
[Spark] Concurrent Identity Columns: sequence-service abstraction + in-memory backend - #7619
Merged
Merged
Conversation
This was referenced Sep 7, 2026
Open
[Spark] Concurrent Identity Columns: DDL integration (CREATE/REPLACE/CONVERT/SYNC/DROP COLUMN)
#7622
Open
Rob2U
changed the title
Rob2U
marked this pull request as ready for review
September 7, 2026 09:46
foss-contributor
added
delta-metadata
mkroll-db
approved these changes
Sep 30, 2026
mkroll-db
left a comment
mkroll-db
left a comment
Collaborator
There was a problem hiding this comment.
LGTM. This is laying the foundation for proper testing and communication.
Draft
1 of 5 tasks
Rob2U
force-pushed
the
stack/cic-export-2-seam
branch
2 times, most recently
from
October 4, 2026 21:52
4fc86d2 to
8ae9038
Compare
murali-db
pushed a commit
that referenced
this pull request
Oct 14, 2026
…or classes (#7618) <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description This PR is the first out of a stack of seven PRs in total. The combined stack adds support for the Concurrent Identity Column (CIC) feature following the RFC [here](#7272). The TLDR for this RFC/feature: It allows for two concurrent 'inserters' to both succeed by adding an identity sequence service that hands out disjoint ranges of id values. This way the two writers do not conflict and can both commit. **_Stack overview:_** 1. Foundation <- this PR (#7618) 2. Service abstraction + In-Memory one (#7619) 3. Transport and buffering for the service requests (#7620) 4. Table feature and the insert path (#7621) 5. DDL integration (#7622) 6. Merge support (#7623) 7. UC Client addition (#7785) This PR introduces no behaviour changes, just scaffolding. It adds: - `ConcurrentIdentityColumnSchema`: the `delta.identity.concurrent.sequenceId` column-metadata key that marks a column as a CIC. - `ConcurrentIdentityColumnErrors` + 8 DELTA_CONCURRENT_IDENTITY_COLUMN_* error classes (disabled, too-many-columns, sequence-not-found, metadata-mismatch, empty-reserve-range, service-timeout, conversion-incomplete, using-wrong-generator). - Config `spark.databricks.delta.identityColumn.concurrent.maxColumnsPerTable` ## How was this patch tested? <!-- If tests were added, say they were added here. Please make sure to test the changes thoroughly including negative and positive cases if possible. If the changes were tested in any way other than unit tests, please clarify how you tested step by step (ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future). If the changes were not tested, please explain why. --> The ConcurrentIdentityColumnErrorsSuite adds three tests for the `EMPTY_RESERVE_RANGE`, `METADATA_MISMATCH`, and `USING_WRONG_GENERATOR` errors. The tests are only added for these errors because these errors can not occur 'naturally' in the code and are therefore tested this way. ## Does this PR introduce _any_ user-facing changes? <!-- If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible. If possible, please also clarify if this is a user-facing change compared to the released Delta Lake versions or within the unreleased branches such as master. If no, write 'No'. --> No --------- Signed-off-by: Robert Weeke <weeke.robert@gmail.com> Co-authored-by: mkroll-db <marco.kroll@databricks.com>
Co-authored-by: mkroll-db <marco.kroll@databricks.com> Signed-off-by: Robert Weeke <weeke.robert@gmail.com>
Signed-off-by: Robert Weeke <weeke.robert@gmail.com>
Rob2U
force-pushed
the
stack/cic-export-2-seam
branch
from
October 14, 2026 18:28
8ae9038 to
0f75ff6
Compare
murali-db
approved these changes
Oct 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which Delta project/connector is this regarding?
Description
This PR is the second out of a stack of seven PRs in total. The combined stack adds support for the Concurrent Identity Column (CIC) feature following the RFC here. TLDR: It allows for two concurrent 'inserters' to both succeed by adding an identity sequence service that hands out disjoint ranges of id values. This way the two writers do not conflict and can both commit.
Stack overview:
([Spark] Foundation for Concurrent Identity Columns: metadata key, error classes #7618)
([Spark] Concurrent Identity Columns: sequence-service abstraction + in-memory backend #7619)
([Spark] Concurrent Identity Columns: executors can reserve ranges from the driver (buffer) #7620)
([Spark] Concurrent Identity Columns: INSERT/append write path and table feature #7621)
([Spark] Concurrent Identity Columns: DDL integration (CREATE/REPLACE/CONVERT/SYNC/DROP COLUMN) #7622)
([Spark] Concurrent Identity Columns: add support for Merge #7623)
([Spark] Concurrent Identity Columns: add client support for the UC Identity Sequence service #7785)
This PR introduces no behaviour changes, just scaffolding. It adds:
createSequence,dropSequence,reserve(+ CreateSequenceRequest / DropSequenceRequest / ReserveIdsRequest / ReserveIdsResponse).IdentitySequenceServices.resolve(spark)for retrieval of the service to be used. Defaults to an in-memoryLocalIdentitySequenceService, overridable viaspark.databricks.delta.identityColumn.concurrent.serviceClassName.How was this patch tested?
The
LocalIdentitySequenceServiceis tested using theLocalIdentitySequenceServiceSuite.scala. These tests ensure that the theLocalIdentitySequenceServicematches the expected behavior exactly. This is important because we will use it later for testing the other components.Does this PR introduce any user-facing changes?
No