Skip to content

added Avro support for KafkaAdapter #645

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
mildek wants to merge 17 commits into
base: main
Choose a base branch
from
Open

added Avro support for KafkaAdapter #645

Show file tree
Hide file tree
Changes from 9 commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
e3ea18f
added Avro support for KafkaAdapter
mildek Dec 4, 2025
8ce67ec
Merge remote-tracking branch 'origin/main' into feat/add-kafka-avro-p…
mildek Dec 9, 2025
7396b22
ruffed scripts
mildek Dec 9, 2025
aa8a87b
conditionally skip avro installation for arm macos
mildek Dec 11, 2025
7eaf32a
fixed bugs, enabled osx-arm wheel
mildek Jan 14, 2026
1f4ab48
fix: linted docs
mildek Jan 14, 2026
26b6b99
fix: add /utf-8 flag for Windows MSVC builds
mildek Jan 14, 2026
cbdb742
fix: add workaround for avro-cpp fmt::formatter incompatibility on Wi…
mildek Jan 14, 2026
63bfb8b
fix: pin fmt to v11.x for avro-cpp compatibility on Windows
mildek Jan 14, 2026
e0d5cd9
fix: workaround for avro-cpp fmt formatter on Windows instead of pinning
mildek Jan 15, 2026
e9d65cb
fix: workaround for avro-cpp fmt formatter on Windows
mildek Jan 15, 2026
acd75a1
fix: patch avro-cpp headers at configure time for fmt v12 compatibili…
mildek Jan 15, 2026
fda0344
fix: address code review feedback
mildek Feb 3, 2026
ec0d6b7
- Extract version from library soname on conda-forge
mildek Feb 5, 2026
d2e8cf7
updated vcpkg sha in setup.py
mildek Feb 5, 2026
f064828
fix: check avro-cpp version instead of code pattern, make Avro optional
mildek Feb 5, 2026
eb68a6e
fix: update vcpkg to get avro-cpp 1.12.1 for fmt v12 compatibility
mildek Feb 5, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -165,7 +165,7 @@ if(WIN32)
add_definitions(-DNDEBUG)
endif()

set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} /EHsc /MP /bigobj")
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} /EHsc /MP /bigobj /utf-8")
Comment thread
AdamGlustein marked this conversation as resolved.
foreach(warning 4244 4251 4267 4275 4290 4786 4305 4996)
SET(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} /wd${warning}")
endforeach(warning)
Expand Down
1 change: 1 addition & 0 deletions conda/dev-environment-unix.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ channels:
- nodefaults
dependencies:
- astor
- avrocpp
Comment thread
timkpaine marked this conversation as resolved.
- bison
- brotli
- bump-my-version
Expand Down
4 changes: 3 additions & 1 deletion conda/dev-environment-win.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,10 @@ channels:
- conda-forge
- nodefaults
dependencies:
# - bison # not available on windows
- astor
- avrocpp
- fmt<12 # Pin fmt to v11.x for avro-cpp compatibility
# - bison # not available on windows
- brotli
- bump-my-version
- cmake
Expand Down
13 changes: 13 additions & 0 deletions cpp/cmake/modules/FindAvro.cmake
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
find_path(Avro_INCLUDE_DIR NAMES avro/Encoder.hh)
find_library(Avro_LIBRARY NAMES avrocpp libavrocpp)

if (NOT TARGET Avro::avrocpp)
add_library(Avro::avrocpp SHARED IMPORTED)
set_property(TARGET Avro::avrocpp PROPERTY
IMPORTED_LOCATION "${Avro_LIBRARY}")
target_include_directories(Avro::avrocpp INTERFACE ${Avro_INCLUDE_DIR})
endif()

include(FindPackageHandleStandardArgs)
find_package_handle_standard_args(Avro DEFAULT_MSG Avro_LIBRARY Avro_INCLUDE_DIR)
mark_as_advanced(Avro_INCLUDE_DIR Avro_LIBRARY Avro::avrocpp)
4 changes: 3 additions & 1 deletion cpp/cmake/modules/FindDepsKafkaAdapter.cmake
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,13 @@ cmake_minimum_required(VERSION 3.7.2)

if (CSP_USE_VCPKG)
find_package(RdKafka CONFIG REQUIRED)
if(NOT WIN32)
find_package(unofficial-avro-cpp CONFIG REQUIRED)
if(NOT WIN32)
# Bad, but a temporary workaround for
# https://github.com/microsoft/vcpkg/issues/40320
link_directories(${VCPKG_INSTALLED_DIR}/${VCPKG_TARGET_TRIPLET}/lib)
endif()
else()
find_package(RdKafka REQUIRED)
find_package(Avro REQUIRED)
endif()
7 changes: 7 additions & 0 deletions cpp/csp/adapters/kafka/CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,13 @@ find_package(DepsKafkaAdapter REQUIRED)

target_link_libraries(csp_kafka_adapter PUBLIC csp_adapter_utils RdKafka::rdkafka RdKafka::rdkafka++)

# Link Avro library
if(CSP_USE_VCPKG)
target_link_libraries(csp_kafka_adapter PUBLIC unofficial::avro-cpp::avrocpp)

@AdamGlustein AdamGlustein Feb 2, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The overall state of avro packaging (both in conda and vcpkg) really gives me hesitation on adding it as a build-time dependency of csp. I'd much prefer if we could split this out into it's own project, but not sure how feasible that is given our adapter ABI is still a WIP.

@mildek mildek Feb 3, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Kafka adapter with Avro support is already fully optional (CSP_BUILD_KAFKA_ADAPTER=ON/OFF). With this PR, on Windows conda-forge builds where avro-cpp is incompatible, it automatically disables itself with a clear warning - no build failures, no patches, just graceful degradation. I agree the avro packaging situation is frustrating. Once the adapter ABI stabilizes, splitting this into a separate project would make sense. For now, keeping it optional within CSP seems like the pragmatic path - users who need Kafka/Avro can enable it, others aren't affected.

@AdamGlustein AdamGlustein Feb 4, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, but I assume we are going to build CSP (for distribution) with Avro enabled, just like we build with Kafka enabled currently. So it's not optional from the perspective of the userm who installs pre-built csp from conda/pypi

@mildek mildek Feb 5, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we wait then for ABI to stabilize before merging it?

@AdamGlustein AdamGlustein Feb 5, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can merge it as an experimental feature and if avro becomes a nuisance in our build pipeline we can always re-evaluate/remove it. @timkpaine what do you think?

@timkpaine timkpaine Feb 5, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we will eventually support separate adapters but for now i think its better to support everything we support on every platform, and not do platform-specific stuff. There should be no incompatible platform as this has caused problems in the past and makes life annoying for users.

else()
target_link_libraries(csp_kafka_adapter PUBLIC Avro::avrocpp)
endif()

install(TARGETS csp_kafka_adapter
PUBLIC_HEADER DESTINATION include/csp/adapters/kafka
RUNTIME DESTINATION ${CSP_RUNTIME_INSTALL_SUBDIR}
Expand Down
3 changes: 3 additions & 0 deletions cpp/csp/adapters/kafka/KafkaPublisher.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
#include <csp/adapters/kafka/KafkaPublisher.h>
#include <csp/adapters/utils/MessageWriter.h>
#include <csp/adapters/utils/JSONMessageWriter.h>
#include <csp/adapters/utils/AvroMessageWriter.h>

#include <librdkafka/rdkafkacpp.h>

Expand All @@ -17,6 +18,8 @@ KafkaPublisher::KafkaPublisher( KafkaAdapterManager * mgr, const Dictionary & pr
auto protocol = properties.get<std::string>( "protocol" );
if( protocol == "JSON" )
m_msgWriter = std::make_shared<utils::JSONMessageWriter>( properties );
else if( protocol == "AVRO" )
m_msgWriter = std::make_shared<utils::AvroMessageWriter>( properties );
else if( protocol != "RAW_BYTES" )
CSP_THROW( NotImplemented, "msg protocol " << protocol << " not currently supported for kafka output adapters" );
}
Expand Down
Loading
Loading