A Dockerized RTSP → Kafka ingestion template with hot-swappable configuration
This repository documents how a fragile, natively running RTSP → Kafka producer was stabilized and productionized through containerization, retry-safe startup, and hot-swappable configuration, without rewriting core logic.
What started as a working Python script on a tower machine evolved into a containerized, operationally resilient producer running reliably on an in-house HPC rack cluster.
This repository is intentionally shared as:
- a learning artifact
- a DevOps reference
- a boilerplate template for teams facing similar ingestion problems
It focuses on operational correctness, not theoretical perfection.
In many data engineering teams, ingestion pipelines begin as native scripts:
- Python scripts using FFmpeg and OpenCV
- Direct Kafka producers
- Manually managed environments
- One machine, one setup, one person who “knows how it works”
That was exactly the case here.
The initial RTSP producer:
- Worked correctly
- Delivered frames to Kafka
- Met functional requirements
Yet, over time, non-functional issues dominated.
The original producer was running natively on a tower machine.
Over time, the following issues emerged:
- Dependency conflicts between system FFmpeg, OpenCV, Python, and Kafka libraries
- Random crashes with no reproducible pattern
- Kafka startup timing causing producer failures
- Manual restarts required during broker restarts
- Environment drift between development and production
- Difficulty moving workloads to rack servers
- No safe way to update configuration without stopping the process
Importantly:
- The code itself was correct
- The failures were operational, not algorithmic
This is a common but under-documented reality in real data pipelines.
During the transition from a native execution model to a containerized setup, the following observable improvements were recorded during internal testing and production trials:
- Producer uptime increased from intermittent (manual restarts required every few hours) to continuous multi-day runs
- Dependency-related failures reduced to zero, once FFmpeg and OpenCV were pinned inside the image
- Mean time to recovery (MTTR) after RTSP or Kafka interruptions dropped from manual intervention to automatic recovery within 2–5 seconds
- Migration from a tower machine to an in-house rack server required no code changes, only a Docker pull and config update
These improvements were achieved without modifying the core ingestion logic, highlighting that most issues were operational rather than algorithmic.
Before changing any code, the following principles were agreed upon:
- Do not rewrite working logic unnecessarily
- Solve operational problems before optimizing performance
- Prefer isolation over clever dependency management
- Make failures explicit and observable
- Keep the system understandable by the next engineer
- Avoid premature orchestration complexity
These principles guided every decision in this repository.
RTSP Camera
↓
FFmpeg (raw frames via pipe)
↓
OpenCV (JPEG encoding)
↓
Kafka Producer
↓
Kafka Topic
This project intentionally focuses only on the producer side.
This repository is versioned intentionally to document the engineering journey.
rtsp-kafka-producer/
├── v1.0-native-container/
│ ├── producer.py
│ ├── requirements.txt
│ ├── Dockerfile
│ ├── docker-compose.yml
│ └── README.md
│
├── v1.1-hotswap-production/
│ ├── producer.py
│ ├── requirements.txt
│ ├── config.yaml.example
│ ├── Dockerfile
│ ├── docker-compose.yml
│ └── README.md
│
└── README.md
└── LICENSE
Each version solves a specific class of problems.
v1.0-native-container/
The goal of v1.0 was not feature expansion.
The goal was to answer a single question:
Can containerization alone stabilize this producer?
- The original working script was placed inside a Docker container
- FFmpeg, OpenCV, and Python dependencies were fixed
- The producer ran in headless mode
- No host bindings
- No runtime config reload
- Eliminated host dependency conflicts
- Removed system-level library cross-talk
- Made restarts deterministic
- Enabled migration from tower machine to rack server
- Allowed the producer to run reliably on an in-house HPC cluster
- Configuration changes required container restarts
- Kafka startup ordering could still cause failures
- No safe runtime tuning
- Operational friction during live changes
v1.0 proved that containerization alone solves most stability problems, but not all.
v1.1-hotswap-production/
The goal of v1.1 was to make the producer:
- operationally flexible
- safe for long-running production
- easy to modify without downtime
- All environment-specific values moved to YAML
- No secrets baked into the image
- Host-mounted configuration directory
- Config file changes detected at runtime
- Internal restart of FFmpeg and Kafka producer
- Container itself remains running
- Downtime limited to seconds
- Kafka connection attempts are retried
- No crash loops when brokers are slow
- Deterministic startup order
- No entrypoint scripts
- No hidden logic
- All behavior visible in logs
v1.1 is the version that ultimately ran smoothly in production on the in-house rack cluster.
This project intentionally does not containerize Kafka.
Reasons:
- Kafka is stateful and storage-heavy
- Broker identity and networking are critical
- Many production Kafka clusters run on VMs or bare metal
Instead, this project demonstrates a practical compromise:
- Keep Kafka external
- Containerize producers and consumers
- Treat Kafka as infrastructure, not an app container
This approach is far more realistic in enterprise environments.
All runtime behavior is controlled via YAML.
Example (safe public version):
rtsp:
url: "rtsp://user:password@camera-ip:554/Streaming/channels/101"
width: 640
height: 480
transport: "tcp"
kafka:
brokers: "localhost:9092"
topic: "camera-stream"
acks: 0
linger_ms: 5
encoding:
jpeg_quality: 95
runtime:
frame_timeout_sec: 5.0
restart_delay_sec: 2To adapt this template:
rtsp.url– camera endpointkafka.brokers– Kafka bootstrap serverskafka.topic– target topic
Optional tuning:
- resolution
- JPEG quality
- timeouts
- restart delay
No code changes required.
- Ensure configuration exists
- Attempt Kafka connection (retry-safe)
- Once Kafka is ready, start FFmpeg
- Begin frame ingestion and publishing
This prevents misleading startup success.
The following characteristics were observed during sustained runtime tests:
- Stable throughput of 18–20 FPS at 640×480 resolution
- JPEG frame sizes typically ranged between 40 KB and 120 KB, depending on scene complexity
- End-to-end producer latency (RTSP → Kafka enqueue) remained within tens of milliseconds under nominal load
- Kafka producer buffers absorbed short broker stalls without crashing the process
It is important to note that performance characteristics are highly dependent on Kafka broker I/O capacity, topic configuration, and replication settings.
- Config file changes
- Change detected
- FFmpeg terminated
- Kafka producer recreated
- Pipeline resumes
Container uptime is preserved.
This producer does not hide Kafka limitations.
High-FPS JPEG ingestion can:
- saturate broker I/O
- cause producer timeouts
- drop frames under pressure
This is expected.
Kafka is a log system, not a media server. This project treats Kafka as an ingestion buffer and fan-out point.
When publishing high-frequency JPEG frames, the following Kafka-side behaviors were observed:
- Under-provisioned brokers resulted in producer-side timeouts after prolonged sustained load
- Frame drops occurred when broker ACKs could not keep pace with ingestion rate
- Increasing FPS exposed Kafka disk and network bottlenecks faster than synthetic message tests
This behavior is expected and highlights an important lesson: real binary payloads reveal infrastructure limits much faster than synthetic benchmarks.
This project reinforced several important truths:
- Most pipeline failures are operational
- Isolation is more valuable than optimization
- Startup ordering matters more than speed
- Explicit failure handling beats silent success
- Simplicity scales better than cleverness
During the two-day iteration cycle, the following failure modes were observed and addressed:
- Kafka startup race conditions causing producer crashes
- Silent frame drops under high broker load
- FFmpeg pipe stalls when RTSP streams paused unexpectedly
- Misleading success logs when Kafka was not yet reachable
Each of these failures directly influenced the final design choices in v1.1, particularly around retry-safe startup and explicit logging.
- Data engineers building ingestion pipelines
- DevOps engineers stabilizing native scripts
- Teams migrating workloads to clusters
- Anyone needing a Kafka producer template
This template may not be suitable if your use case requires:
- Guaranteed delivery of every video frame
- Frame-perfect ordering across all consumers
- Ultra-low-latency media streaming to end users
- Media playback or real-time viewing workloads
- Strong transactional guarantees on video payloads
Kafka is used here as an ingestion and buffering layer, not as a media transport. Frame drops under sustained load are expected and acceptable in this design.
If your requirements include strict delivery guarantees or real-time playback, consider dedicated video streaming or media transport protocols instead.
- Not a video streaming platform
- Not a Kafka consumer implementation
- Not guaranteed delivery
- Not a media playback system
This repository is shared to:
- Document a real engineering journey
- Help others avoid similar pitfalls
- Provide a reusable production template
- Show how incremental improvements matter
If this helps someone stabilize their own pipeline, it has served its purpose.
This repository intentionally favors clarity over brevity.
The goal is not to impress, but to teach and document how real systems are built, tested, broken, and fixed.
If you adapt this template, modify it to fit your constraints. There is no single correct architecture, only context-appropriate ones.
This project was intentionally developed over a short iteration window (approximately two days).
The focus was not feature completeness, but:
- isolating real failure modes
- validating assumptions under load
- incrementally improving operational reliability
This mirrors how many in-house production systems evolve under real constraints.