Conversation
…sion options. Set compatibility with 2.x client by default. Update tests
There was a problem hiding this comment.
Pull request overview
Updates the Snowplow Enrich Kinesis/EventBridge modules to use amazon-kinesis-client 3.5.0 (and a newer AWS SDK), introducing a new input setting to control KCL 2.x→3.x migration compatibility mode and wiring that through to the KCL scheduler configuration. This aligns the codebase with KCL 3.5’s single-table migration behavior and documents the operational rollout steps.
Changes:
- Bump
amazon-kinesis-clientto3.5.0(and AWS SDK to2.47.5). - Add
clientVersionConfigtoInput.Kinesisconfig model, config templates, and moduleapplication.conf. - Wire
clientVersionConfiginto KCLCoordinatorConfigand update unit/integration tests and add a migration guide.
Reviewed changes
Copilot reviewed 14 out of 14 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| project/Dependencies.scala | Bumps AWS SDK and KCL dependency versions. |
| modules/common-fs2/src/main/scala/com/snowplowanalytics/snowplow/enrich/common/fs2/config/io.scala | Adds clientVersionConfig to Input.Kinesis plus Circe encoder/decoder. |
| modules/kinesis/src/main/scala/com/snowplowanalytics/snowplow/enrich/kinesis/Source.scala | Applies clientVersionConfig to KCL coordinator configuration before building the Scheduler. |
| modules/eventbridge/src/main/scala/com/snowplowanalytics/snowplow/enrich/eventbridge/Source.scala | Same coordinator configuration wiring for the EventBridge module. |
| modules/eventbridge/src/it/scala/com/snowplowanalytics/snowplow/enrich/kinesis/Source.scala | Mirrors the coordinator configuration wiring for integration tests. |
| modules/kinesis/src/main/resources/application.conf | Sets default input.clientVersionConfig to COMPATIBLE_WITH_2X_PHASE1. |
| modules/eventbridge/src/main/resources/application.conf | Sets default input.clientVersionConfig to COMPATIBLE_WITH_2X_PHASE1. |
| config/config.kinesis.extended.hocon | Documents and includes clientVersionConfig in the example config. |
| config/config.eventbridge.extended.hocon | Documents and includes clientVersionConfig in the example config. |
| modules/kinesis/src/test/scala/com/snowplowanalytics/snowplow/enrich/kinesis/ConfigSpec.scala | Updates expected parsed config to include clientVersionConfig. |
| modules/eventbridge/src/test/scala/com/snowplowanalytics/snowplow/enrich/eventbridge/ConfigSpec.scala | Updates expected parsed config to include clientVersionConfig. |
| modules/kinesis/src/it/scala/com/snowplowanalytics/snowplow/enrich/kinesis/KinesisConfig.scala | Updates integration-test config constructors with clientVersionConfig. |
| modules/eventbridge/src/it/scala/com/snowplowanalytics/snowplow/enrich/eventbridge/IntegrationTestConfig.scala | Updates integration-test config constructors with clientVersionConfig. |
| docs/KCL_3_MIGRATION.MD | Adds a detailed operational migration/rollout document for maintainers. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+147
to
+151
| val coordinatorConfig = configsBuilder.coordinatorConfig | ||
| .clientVersionConfig { | ||
| kinesisConfig.clientVersionConfig match { | ||
| case Input.Kinesis.ClientVersionConfig.CompatibleWith2xPhase1 => | ||
| CoordinatorConfig.ClientVersionConfig.CLIENT_VERSION_CONFIG_COMPATIBLE_WITH_2X_PHASE1 |
Comment on lines
+140
to
+144
| val coordinatorConfig = configsBuilder.coordinatorConfig | ||
| .clientVersionConfig { | ||
| kinesisConfig.clientVersionConfig match { | ||
| case Input.Kinesis.ClientVersionConfig.CompatibleWith2xPhase1 => | ||
| CoordinatorConfig.ClientVersionConfig.CLIENT_VERSION_CONFIG_COMPATIBLE_WITH_2X_PHASE1 |
Comment on lines
+147
to
+151
| val coordinatorConfig = configsBuilder.coordinatorConfig | ||
| .clientVersionConfig { | ||
| kinesisConfig.clientVersionConfig match { | ||
| case Input.Kinesis.ClientVersionConfig.CompatibleWith2xPhase1 => | ||
| CoordinatorConfig.ClientVersionConfig.CLIENT_VERSION_CONFIG_COMPATIBLE_WITH_2X_PHASE1 |
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
oermolaev
marked this pull request as ready for review
July 31, 2026 15:26
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
used from day one, elimination the need for complex schema migrations.
clientVersionConfig, was added toInput.Kinesisand mululesapplication.conf.COMPATIBLE_WITH_2X_PHASE1)