Skip to content

Upgrade kinesis client to v3.5.0. - #350

Open
oermolaev wants to merge 5 commits into
mainfrom
kcl-v3
Open

Upgrade kinesis client to v3.5.0.#350
oermolaev wants to merge 5 commits into
mainfrom
kcl-v3

Conversation

@oermolaev

@oermolaev oermolaev commented Jul 24, 2026

Copy link
Copy Markdown
Collaborator
  • Kinesis client library is set to 3.5.1 version. This way, for direct 2.x -> 3.5 migrations, single DynamoDB table format is
    used from day one, elimination the need for complex schema migrations.
  • Modules are affected by this update: kinesis and eventbridge
  • A new input config field, clientVersionConfig, was added to Input.Kinesis and mulules application.conf.
  • Compatibility with 2.x client is set by default (COMPATIBLE_WITH_2X_PHASE1)
  • Tests updated
  • Migration documentation added; infrastructure update notes & checklist included

…sion options. Set compatibility with 2.x client by default. Update tests
@oermolaev oermolaev self-assigned this Jul 24, 2026
@oermolaev oermolaev added the enhancement New feature or request label Jul 24, 2026
@oermolaev oermolaev changed the title Upgrade kinesis client to v3.50. Extend configuration with client ver… Upgrade kinesis client to v3.5.0. Extend configuration with client ver… Jul 24, 2026
@oermolaev oermolaev changed the title Upgrade kinesis client to v3.5.0. Extend configuration with client ver… Upgrade kinesis client to v3.5.0. Jul 24, 2026

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Updates the Snowplow Enrich Kinesis/EventBridge modules to use amazon-kinesis-client 3.5.0 (and a newer AWS SDK), introducing a new input setting to control KCL 2.x→3.x migration compatibility mode and wiring that through to the KCL scheduler configuration. This aligns the codebase with KCL 3.5’s single-table migration behavior and documents the operational rollout steps.

Changes:

  • Bump amazon-kinesis-client to 3.5.0 (and AWS SDK to 2.47.5).
  • Add clientVersionConfig to Input.Kinesis config model, config templates, and module application.conf.
  • Wire clientVersionConfig into KCL CoordinatorConfig and update unit/integration tests and add a migration guide.

Reviewed changes

Copilot reviewed 14 out of 14 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
project/Dependencies.scala Bumps AWS SDK and KCL dependency versions.
modules/common-fs2/src/main/scala/com/snowplowanalytics/snowplow/enrich/common/fs2/config/io.scala Adds clientVersionConfig to Input.Kinesis plus Circe encoder/decoder.
modules/kinesis/src/main/scala/com/snowplowanalytics/snowplow/enrich/kinesis/Source.scala Applies clientVersionConfig to KCL coordinator configuration before building the Scheduler.
modules/eventbridge/src/main/scala/com/snowplowanalytics/snowplow/enrich/eventbridge/Source.scala Same coordinator configuration wiring for the EventBridge module.
modules/eventbridge/src/it/scala/com/snowplowanalytics/snowplow/enrich/kinesis/Source.scala Mirrors the coordinator configuration wiring for integration tests.
modules/kinesis/src/main/resources/application.conf Sets default input.clientVersionConfig to COMPATIBLE_WITH_2X_PHASE1.
modules/eventbridge/src/main/resources/application.conf Sets default input.clientVersionConfig to COMPATIBLE_WITH_2X_PHASE1.
config/config.kinesis.extended.hocon Documents and includes clientVersionConfig in the example config.
config/config.eventbridge.extended.hocon Documents and includes clientVersionConfig in the example config.
modules/kinesis/src/test/scala/com/snowplowanalytics/snowplow/enrich/kinesis/ConfigSpec.scala Updates expected parsed config to include clientVersionConfig.
modules/eventbridge/src/test/scala/com/snowplowanalytics/snowplow/enrich/eventbridge/ConfigSpec.scala Updates expected parsed config to include clientVersionConfig.
modules/kinesis/src/it/scala/com/snowplowanalytics/snowplow/enrich/kinesis/KinesisConfig.scala Updates integration-test config constructors with clientVersionConfig.
modules/eventbridge/src/it/scala/com/snowplowanalytics/snowplow/enrich/eventbridge/IntegrationTestConfig.scala Updates integration-test config constructors with clientVersionConfig.
docs/KCL_3_MIGRATION.MD Adds a detailed operational migration/rollout document for maintainers.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +147 to +151
val coordinatorConfig = configsBuilder.coordinatorConfig
.clientVersionConfig {
kinesisConfig.clientVersionConfig match {
case Input.Kinesis.ClientVersionConfig.CompatibleWith2xPhase1 =>
CoordinatorConfig.ClientVersionConfig.CLIENT_VERSION_CONFIG_COMPATIBLE_WITH_2X_PHASE1
Comment on lines +140 to +144
val coordinatorConfig = configsBuilder.coordinatorConfig
.clientVersionConfig {
kinesisConfig.clientVersionConfig match {
case Input.Kinesis.ClientVersionConfig.CompatibleWith2xPhase1 =>
CoordinatorConfig.ClientVersionConfig.CLIENT_VERSION_CONFIG_COMPATIBLE_WITH_2X_PHASE1
Comment on lines +147 to +151
val coordinatorConfig = configsBuilder.coordinatorConfig
.clientVersionConfig {
kinesisConfig.clientVersionConfig match {
case Input.Kinesis.ClientVersionConfig.CompatibleWith2xPhase1 =>
CoordinatorConfig.ClientVersionConfig.CLIENT_VERSION_CONFIG_COMPATIBLE_WITH_2X_PHASE1
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
@oermolaev
oermolaev marked this pull request as ready for review July 31, 2026 15:26
@oermolaev
oermolaev requested a review from joaolcorreia July 31, 2026 15:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants