A stream-batch unified observability lakehouse for LLM and Agent applications. It models requests, agent runs, retrievals, feedback, safety, evaluation, model releases, compliance, multi-agent orchestration and platform health events into traceable DWD facts, reusable DWS metrics and application-facing ADS data products.
- Cost — how many tokens and how much budget do models, features, teams and applications consume?
- Reliability — which requests, agents, tools or handoffs fail or time out most often?
- Performance — is end-to-end latency spent in the model, retrieval, tools or agent orchestration?
- Quality — how do user feedback, offline evaluation, prompt versions and retrieval quality change over time?
- Governance — are sensitive-resource access, retention policies and guardrails enforced as specified?
- Platform — are Kafka, Flink, Paimon and Doris health metrics crossing their thresholds?
flowchart LR
S["Apps / DeepSeek / Hermes / Mock / Enterprise systems"] --> R["Raw JSONL / Postgres / Kafka"]
R --> K["Kafka ODS"]
K --> F["Flink SQL"]
R --> B["Spark backfill & validation"]
F --> P["Paimon DWD / DWS"]
B --> P
G["Gravitino Metalake / Catalog"] -.metadata management.-> P
P --> D["Doris query & serving layer"]
D --> U["Superset analytics / Grafana monitoring"]
Core components: Python 3.11, Postgres 16, Kafka 3.9, Flink 1.20, Apache Paimon, Spark 3.5, Apache Gravitino 1.2, Doris 2.1, Superset and Grafana. Gravitino manages and exposes the shared Paimon catalog metadata through the ai_observability metalake; Paimon still owns data files and snapshots, and compute engines keep reading and writing through the Paimon runtime. See Architecture for full component boundaries and data flow.
The current Doris data product catalog contains 46 tables:
| Layer | Count | Coverage |
|---|---|---|
| DWD | 12 | LLM, agent, retrieval, feedback, safety, evaluation, release, compliance, orchestration |
| DWS | 16 | Daily/hourly/session, team/region/environment, quality, cost, health metrics |
| DIM | 7 | Model, model version, prompt, team, user, knowledge base, rule |
| ADS | 11 | Cost, SLA, quality, safety, satisfaction, trace health, evaluation dataset/experiment regression, executive weekly summary |
The default automated Postgres CDC entry point currently covers LLM requests only. The other domains ship event models, Kafka ODS/Flink, Spark, Paimon/Doris assets and tests, but production producers need to emit events to the corresponding Kafka topics or batch entry points.
Prerequisites: Docker Compose, uv, and at least 12 GB of available memory. For day-to-day development, start the lightweight path first:
make infra-lightmake seed-datamake flink-submitscripts/check_pipeline_health.sh --skip-servingBring up the serving components when you need the query layer and dashboards:
make infra-servingmake sync-dorismake infra-dashboardmake init-supersetmake healthLocal endpoints: Flink http://localhost:8081, Gravitino Web V2 http://localhost:8090, Doris FE http://localhost:8030, Superset http://localhost:8088, Grafana http://localhost:3001. Check Gravitino status and catalogs with make gravitino-status and make gravitino-catalogs. Default demo credentials are listed in the Runtime Runbook — never reuse them in shared or production environments.
Run the tests:
uv run pytest -v| Document | Purpose |
|---|---|
| Product Document | Users, scenarios, capability scope, data products and non-goals |
| Architecture | Component boundaries, data flow, deployment topology and key design decisions |
| Technical Document | Modules, contracts, processing semantics, quality and extension points |
| Data Model | Layering, table inventory, grain, key relations and metric constraints |
| Metric Definitions | Metric semantics and how to query them |
| Data Lineage | Domain-to-table, job and consumer lineage |
| Runtime Runbook | Start/stop, checks, recovery and resource guidance |
| Flink Jobs | SQL execution order and runtime notes |
| Architecture Decision Records | Paimon, Kafka, DWS, DQ and dashboard decisions |
| Full documentation index | Current state, reports, ADRs and historical plans |
docs/*_plan.md and the migration plans record how the project evolved and do not describe pending scope. The current state is defined by this README, the architecture/technical/product documents, the code, the DDL and the tests.
app/ Event models, data contracts, quality rules and shared logic
scripts/ Ingestion, generation, Spark jobs, loading and operations scripts
flink/sql/ Flink catalog, ODS, DWD, DWS and verification SQL
sql/ Postgres source tables, Doris DDL, sync and query SQL
config/ SLA, platform thresholds, Superset/Grafana assets
docs/ Product, architecture, technical, model, lineage, runbook and ADRs
tests/ Unit, contract, SQL asset and optional Paimon integration tests
Please read AGENTS.md before contributing or changing data assets.