Serverless MSK to External Webhook Pipeline

This project demonstrates a production-grade, serverless architecture using AWS EventBridge Pipes to securely stream real-time data from a completely private Amazon Managed Streaming for Apache Kafka (MSK) cluster to an external, 3rd-party HTTP Webhook.

Architecture Overview

Traditionally, consuming data from a private MSK cluster requires provisioning and maintaining self-hosted EC2 runners or containerized consumers inside the VPC. This architecture eliminates that compute overhead entirely.

flowchart LR
    subgraph AWS Cloud
        subgraph Private VPC
            MSK[(Amazon MSK\nTopic)]
            ENI((VPC ENI))
        end
        Pipe{EventBridge\nPipe}
        Auth[EventBridge\nConnection]
        Dest[API\nDestination]
    end
    
    Webhook[External 3rd-Party\nWebhook/Supabase]

    MSK -->|Reads messages| ENI
    ENI --> Pipe
    Auth -->|Injects API Key| Dest
    Pipe -->|BatchSize: 1| Dest
    Dest -->|HTTPS POST| Webhook

Loading

Key Technical Challenges Solved

1. The VPC Networking Gap

Problem: EventBridge Pipes run on AWS-managed compute in an isolated service network. It cannot natively reach a private MSK cluster inside a custom VPC. Solution: We attached ec2:CreateNetworkInterface permissions to the IAM execution role. This allows EventBridge to dynamically provision Hyperplane Elastic Network Interfaces (ENIs) directly inside the private subnets, establishing a secure, private bridge to the Kafka brokers.

2. External API Rate Limiting & Batching

Problem: External endpoints can be easily overwhelmed (DDoS'd) if MSK blasts millions of backfill records instantly. Furthermore, EventBridge API Destinations enforce a strict BatchSize: 1 limit for HTTP targets. Solution: We configured the API Destination with a hard limit of 300 requests per second. The Pipe natively throttles the MSK consumer lag, ensuring a safe, steady stream of individual HTTP POST requests without breaking the destination server.

Deployment (Infrastructure as Code)

This entire pipeline was provisioned exclusively via the AWS CLI. See setup.sh for the exact execution commands used to build the Connection, API Destination, IAM Roles, and EventBridge Pipe.