An experimental lab for production failure-paths in event-driven microservices: transactional outbox, DLQ parking, manual offset commits, KRaft Kafka, Kustomize deploys, scanned GHCR images — all runnable locally with one command.
The
mahasiswa(student records) domain is demo data on purpose. The patterns are the product. For backend devs who outgrew hello-world Kafka tutorials — not a real SIS (no auth, no multi-tenancy), not a product, not a template. Read the releases as the checklist: v2 security, v3 platform, v4 reliability, v5 operations. That checklist IS the urgency — tutorials teach the happy path; production lives everywhere else.
This repo is an npm workspaces monorepo. Shared code lives in
packages/common (@kaf kube/common).
flowchart TD
FE[mahasiswa-app-frontend<br/>React + Vite + Nginx] -->|/api| API[rest-api-mahasiswa<br/>Express + @kaf kube/common]
API -->|txn: row + outbox| PGMAIN[(PostgreSQL<br/>mahasiswa_db)]
RELAY[outbox relay<br/>advisory-locked poll] -->|SELECT unpublished| PGMAIN
RELAY -->|MAHASISWA_* events| KAFKA[(Kafka KRaft)]
KAFKA --> CONS[consumer-service<br/>manual commit + DLQ]
CONS -->|SQL| PGLOG[(PostgreSQL<br/>log_db)]
CONS -->|/metrics :9400| PROM[Prometheus]
PROM --> GRAF[Grafana]
DASH[dashboard-frontend] -->|PromQL| PROM
| Component | Technology | Role |
|---|---|---|
| REST API | Node.js, Express, pg, kafkajs, @kaf kube/common | Writes rows + outbox events atomically, relay publishes to Kafka |
| Main Database | PostgreSQL 15 | Master data + transactional outbox for student entities |
| Message Broker | Apache Kafka (KRaft, no Zookeeper) | Distributed event log for decoupling services |
| Consumer Service | Node.js, kafkajs, prom-client, @kaf kube/common | Manual offset commit, persists audit logs, parks poison in DLQ, exports metrics |
| Log Database | PostgreSQL 15 | Dedicated storage for audit trails and latency metrics |
| Web Frontend | React, Vite, Tailwind CSS, Nginx | Student management UI with interactive form |
| Dashboard | Node.js, Express, Prometheus API | Cluster-level health and node metrics viewer |
| Monitoring | Prometheus, Grafana, Kube-State-Metrics | Real-time observability and latency dashboard |
Make sure Docker and Docker Compose are installed and running.
# Clone the repository
git clone https://github.com/cipheraenys/Kafkube-Experimental.git
cd Kafkube-Experimental
# Copy env template and fill secrets (never commit .env)
cp .env.example .env
# Start all services (Kafka, PostgreSQL DBs, API, Consumer, Frontend)
docker compose up --buildAccess services locally:
- Web Frontend: http://localhost:8080
- REST API: http://localhost:3000 (
/health,/api/mahasiswa) - Consumer Metrics: http://localhost:9400/metrics
- Kafka Host Listener:
localhost:29092 - Main PostgreSQL:
localhost:5432 - Log PostgreSQL:
localhost:5433
| Variable | Used by | Default | Notes |
|---|---|---|---|
PORT |
rest-api | 3000 |
HTTP listen port |
DATABASE_URL |
rest-api | — (required) | Secret app-secrets in K8s, .env in compose |
LOG_DATABASE_URL |
consumer | — (required) | Secret app-secrets in K8s, .env in compose |
KAFKA_BROKERS |
rest-api, consumer | kafka:9092 (compose) / kafka-service:9092 (K8s) |
Comma-separated list |
KAFKA_TOPIC |
rest-api, consumer | mahasiswa_events |
DLQ topic is <topic>.DLQ |
KAFKA_CLIENT_ID |
rest-api, consumer | mahasiswa-api-producer / random consumer-service-* |
Override per replica if needed |
KAFKA_CONSUMER_GROUP |
consumer | log_processors_group |
Compose/K8s override per environment |
KAFKA_CONSUME_FROM_BEGINNING |
consumer | "true" |
Set "false" in prod to avoid replay |
KAFKA_DLQ_TOPIC |
consumer | <KAFKA_TOPIC>.DLQ |
Poison messages park here with error header |
OUTBOX_POLL_MS |
rest-api | 1000 |
Outbox relay poll interval |
METRICS_PORT |
consumer | 9400 |
Prometheus scrape port |
LOG_LEVEL |
rest-api, consumer | info |
pino log level (debug for verbose) |
CORS_ORIGINS |
rest-api | allow all when empty | Comma-separated allowlist for prod |
VITE_API_URL |
mahasiswa-frontend (build) | /api |
Baked at vite build time |
API_URL |
latency-test, k6 load-test | http://localhost:3000/api/mahasiswa |
Target for benchmarks |
LOG_DATABASE_URL |
latency-test | port 5433 variant |
Used by tests/latency-test.sh port-forward flow |
PROMETHEUS_URL |
dashboard | Prometheus in-cluster URL | Injected via dashboard-config ConfigMap |
Local dev without Docker: nvm use (Node >= 22.12, see .nvmrc), npm ci at root, then per-service npm run dev / vite.
Manifests live in k8s/: base/{mahasiswa-app,dashboard,monitoring,streaming,databases,security}/
plus the overlays/production/ entrypoint. Always deploy via the overlay (it remaps
images to GHCR); never kubectl apply -f a base directory directly.
Kafka runs on Strimzi and Postgres on CloudNativePG — both installed from pinned
bundles by scripts/install-operators.sh (runs first inside deploy-all.sh;
verify the pinned versions against upstream releases before upgrading).
Public ingresses terminate self-signed TLS via cert-manager; kafka-ui additionally
requires basic-auth (secret-kafka-ui-auth.yaml, same gitignored pattern as
secrets.yaml). pgAdmin keeps its own login form.
Liveness probes hit /health, readiness probes hit /ready (checks database
connectivity; Kafka stays out of readiness on purpose since delivery is
decoupled through the outbox). An optional KEDA lag scaler lives in
k8s/optional/keda/ (see its README; not applied by default).
# 1. Create the gitignored Secret from the template
cp k8s/base/mahasiswa-app/secrets.example.yaml k8s/base/mahasiswa-app/secrets.yaml
# edit secrets.yaml with real values (never commit it)
# 2. Deploy everything
cd scripts/
./deploy-all.shcd scripts/
./delete-all.sh --yesRelease images are built by .github/workflows/images.yml and pushed to
ghcr.io/cipheraenys/{rest-api,consumer-service,frontend,dashboard}:<semver>-<sha>.
The production overlay pins them via the images: transformer.
# Unit tests for every workspace (api, consumer, common, frontend, dashboard, latency syntax)
npm run test --workspaces --if-present
# Frontend only (Vitest)
npm run test --workspace=mahasiswa-frontend
# Lint frontend
npm run lint --workspace=mahasiswa-frontendAdditional suites in tests/:
- Horizontal Scaling Test:
cd tests/load-test API_URL=http://localhost:3000/api/mahasiswa ./send-events.sh - Latency Benchmarking (needs a running cluster):
cd tests ./latency-test.sh
- v1 → v2 (breaking): credentials moved from ConfigMap/inline values to
Secret app-secrets(K8s) and.env(compose); Node 18 → 22; Kafka stays single-broker. Old passwords are compromised — rotate, do not reuse. - v2 → v3 (breaking): npm workspaces (single root lockfile,
npm ciat root);packages/commonshared lib;kubernetes/moved tok8s/with semantic filenames + Kustomize (kubectl apply -k k8s/overlays/production); images moved toghcr.io/cipheraenys/*; Kafka runs KRaft (Zookeeper removed); Postgres runs as StatefulSet. - v3 → v4 (breaking): writes go through a transactional outbox polled by
an advisory-locked relay, so responses no longer carry
warning: event_pending; consumer commits offsets manually per batch (at-least-once + DLQ, no poison loops); readiness moved to/ready.
k8s/base/mahasiswa-app/secrets.yaml not found— copy fromsecrets.example.yamland fill values; the file stays local-only.EADDRINUSEonnpm run dev— Vite dev uses port 3001, API uses 3000; change viavite.config.js/PORT.- Consumer lag / poison messages — check consumer logs and consume
<KAFKA_TOPIC>.DLQto inspect parked messages with their error headers. Stuck unpublished events:SELECT id, attempts FROM outbox WHERE published_at IS NULL ORDER BY id;(relay interval:OUTBOX_POLL_MS). - Wrong Node version — run
nvm use(repo pins Node >= 22.12 via.nvmrcandengines). - CloudNativePG passwords — the
approle password is generated into the<cluster>-appsecret at cluster creation; copy it intosecrets.yaml(DATABASE_URL/LOG_DATABASE_URL) before starting the API and consumer:kubectl get secret postgres-main-app -n mahasiswa-app -o jsonpath='{.data.password}' | base64 -d(same forpostgres-log-app). Backups: schedule VolumeSnapshots or wire an S3 store; see the CNPG docs for your cluster before relying on single replicas.
Contributions are welcome. See CONTRIBUTING.md for guidelines and development workflow.
Special thanks to riskyprsty for contributions, support, and inspiration throughout the development of this project.
This project is licensed under the MIT License.