Turn your e-commerce and self-media platform data into clean dashboards, customer analytics, and natural-language queries — self-hosted, zero scraping.
You sell on platforms like Youzan, JD, and Tmall, and you create content on WeChat Official Accounts, Xiaohongshu (XHS), and Zhihu. Each platform has data you need — but the formats are all different. Dozens of differently-named columns. "Customer" doesn't mean the same thing across platforms. Some metrics are cumulative. The same order might appear in multiple rows and needs de-duplication.
General-purpose BI tools (Tableau, Metabase, etc.) won't handle this for you. They draw charts on whatever schema you hand them — whether the numbers are right is your problem.
OmniPanel handles the correctness layer for you. How data gets in depends on the platform:
- Platforms with an API (WeChat Official Accounts) — configure once, background sync pulls fresh data daily
- Platforms with a creator portal (XHS, Zhihu, and XHS's Pugongying KOL/KOC collaboration platform) — a built-in collector agent logs in and exports automatically
- Export-only platforms (Youzan, JD, Tmall orders) — upload the spreadsheet, auto-detected by column fingerprint
Regardless of how it arrives, the data is automatically:
- 🏷️ Identified — column fingerprinting detects which platform each dataset came from
- 🔄 Normalized — mapped onto one unified schema (customer, amount, date, region, etc.)
- 📊 Analyzed — dashboards, customer profiles, cohort retention, cross-platform identity resolution, a SQL console, and natural-language queries
All data comes through official channels (official APIs, official export features) — no scraping, no grey areas.
Data arrives through two paths; the analytics experience is the same either way:
Path A: Auto-sync Path B: Manual upload
(WeChat OA / XHS / Zhihu) (Youzan / JD / Tmall orders)
│ │
└──────────┬─────────────────────────┘
▼
OmniPanel detects → normalizes → stores
▼
Dashboards · Customer analytics · SQL console · NL queries
- Auto-sync: configure a platform account once; the background job pulls fresh data daily with no manual steps
- Manual upload: export a file from the platform's back-office, drop it into OmniPanel — auto-detected and stored (content-hash de-duplication means re-uploads are safe)
- Analyze: open dashboards, write SQL in the console, or ask questions in plain Chinese ("what was the repurchase rate in Guangdong last month?")
All data shown is randomly generated; not a real business.
| Customer analytics | Cohort retention |
|---|---|
 |
 |
| Cross-platform customer identity | SQL console |
|---|---|
 |
 |
| Type | Platforms | How |
|---|---|---|
| E-commerce orders | Youzan, JD, Tmall | Upload official exports (.xlsx/.xls/.csv); auto-detected by column fingerprint |
| WeChat OA | WeChat Official Accounts | Automatic API sync (daily scheduled pull), no manual steps needed |
| Content platforms | Xiaohongshu (XHS), Zhihu | Upload official exports or use the built-in automated collector agent |
| KOL/KOC collaboration | Pugongying (蒲公英, XHS's official creator-collaboration platform) | Upload official campaign exports or use the same collector agent |
Content-hash de-duplication means re-uploading the same file is safe — no duplicates.
- Customer overview — new vs. returning, repurchase rate and time-to-repurchase, per-customer order history, regional distribution
- Cohort retention — monthly cohorts with right-censored retention curves
- Order browser — searchable, exportable view of all normalized orders
- Cross-platform customer identity — merge the same person's orders across Youzan, JD, and Tmall by phone number, with explicit exact/fuzzy confidence tiers (JD masks phone numbers; the fuzzy tier uses partial fingerprint matching and is structurally separate from the exact tier)
- WeChat traffic — daily reads, shares, followers; trends and comparisons
- Content-to-sales impact — correlate publish dates with order volume to measure content-driven sales, across WeChat, XHS, and Zhihu
- Topic-tag diagnostics — keyword-based content-topic tagging (XHS, Zhihu) for spotting which subjects actually drive engagement
- KOL/KOC collaboration analytics (Pugongying) — a 7-view dashboard covering blogger- and campaign-level ROI (spend, impressions, reads, engagement rate), audience demographics (age/gender/device/region/interest breakdowns), and component-level click performance (text links, bottom bar, interactive stickers, comment-area links)
- SQL console — read-only ad-hoc queries with strict guardrails: SELECT/WITH only, auto
LIMITinjection, statement timeout, full audit logging. Save and share frequently-used queries. - Natural-language queries (NL-to-SQL) — ask questions in plain Chinese and get back generated SQL + results. Pluggable LLM provider support (Anthropic, OpenAI, MiniMax, DeepSeek, Moonshot, Zhipu); API keys stay server-side and users pick provider/model from a dropdown.
- Three roles — viewer (read-only) / analyst (upload + analyze) / admin (user management, DB operations)
- Enterprise WeChat SSO — optional QR-code login
- Audit log — every query and mutating action is written to an append-only operation log
- Background jobs — scheduled WeChat metric sync and monthly DB backups, leader-elected for multi-worker safety
- Watchdog — daily health check on every background pipeline (WeChat sync, collector, backups); alerts via WeCom if one stops running altogether, on top of per-run success/failure notifications
- Proactive session health check — the collector's
verify-allchecks every saved creator-portal login on its own, earlier schedule, so an expired session is caught with lead time to fix it instead of discovered mid-run - Granular alert routing — admins pick exactly who receives WeCom alerts, per user, from the user management page — no
.envedits needed
Scraper-based tools have three fundamental problems:
- Legal risk — they operate in a grey area, and platform ToS policies vary
- Fragility — they break every time a platform redesigns its pages or tightens anti-bot measures
- Maintenance burden — you need to track frontend changes across every platform
OmniPanel only pulls data through official channels — APIs where available (WeChat OA), official creator-portal exports where offered (XHS, Zhihu), and platform-provided export features otherwise (e-commerce orders). These are authorized, structurally stable data sources that don't break on frontend redesigns. It spends its effort on the correctness layer that generic BI tools skip, instead of fighting anti-bot systems.
How this compares to similar projects
| Project | Data source | What it ships |
|---|---|---|
| OmniPanel (this repo) | Official channels: APIs + official exports (Youzan/JD/Tmall, WeChat OA/XHS/Zhihu) | Self-hosted app: dashboards, cohort/identity analytics, SQL console, NL-to-SQL |
| DA_Multi_Agent_Workflow | Platform APIs + crawlers (Douyin Shop, XHS, WeChat Channels, ad platforms) | n8n-orchestrated multi-agent ETL/analytics pipeline synced into Feishu |
| ECommerceCrawlers | Web scraping (Taobao, Xianyu, Weibo, 20+ sites) | Scraper code samples; not a deployable product |
| data-api (Just One API) | Web scraping, 40+ platforms | Hosted pay-per-call data feed; no analytics layer |
| bodapi global-ecommerce-data-scraping-solutions-cn | Web scraping with anti-bot bypass, 20+ global platforms | Hosted cross-border data service |
See docs/comparison.md for a detailed breakdown.
Browser (Streamlit, :8501) ──→ Backend API (FastAPI, :8000) ──→ PostgreSQL
│
┌────┴────┐
│ Redis │ (optional, cache / rate limiting)
│ LLM API │ (optional, NL-to-SQL)
└──────────┘
| Layer | Technology | Role |
|---|---|---|
| Frontend | Streamlit (app/ui/) |
Thin client — renders backend data; no business logic |
| Backend | FastAPI (app/) |
Auth, ETL, analytics, SQL console, background jobs |
| Database | PostgreSQL + SQLAlchemy | Unified normalized schema; raw platform rows preserved alongside |
| Cache | Redis (optional) | Distributed cache and login rate limiter; falls back to in-process |
See Architecture for full details.
Requires Docker + Docker Compose v2. Starts Postgres, FastAPI, and Streamlit together — no local Python or Postgres needed.
git clone https://github.com/Nanboy-Ronan/OmniPanel.git
cd OmniPanel
cp .env.example .env
# edit .env: set RAP_SECRET to a strong random value
# python -c "import secrets; print(secrets.token_urlsafe(48))"
docker compose up --buildOpen http://localhost:8501, register the first user (auto-promoted to admin), and start uploading exports.
Requires Python 3.13+ and PostgreSQL 13+.
# 1. Clone and install
git clone https://github.com/Nanboy-Ronan/OmniPanel.git
cd OmniPanel
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
# 2. Configure
cp .env.example .env
# edit .env: set RAP_DATABASE_URL, RAP_SECRET, and (optionally) an LLM API key
# 3. Apply database migrations
make db-upgrade # or: alembic upgrade head
# 4. Start the backend (FastAPI on :8000)
uvicorn app.main:app --host 0.0.0.0 --port 8000
# 5. In another terminal, start the frontend (Streamlit on :8501)
streamlit run app/ui/dashboard.pyFull walkthrough: Getting started.
All settings are environment variables (full list in .env.example).
| Variable | Purpose |
|---|---|
RAP_DATABASE_URL |
PostgreSQL connection string |
RAP_SECRET |
JWT signing key — must be changed; generate with python -c "import secrets; print(secrets.token_urlsafe(48))" |
RAP_SECRET_PREVIOUS |
Previous RAP_SECRET values (comma-separated), used to validate old tokens during key rotation |
FORWARDED_ALLOW_IPS |
Trusted proxy IPs when behind a reverse proxy (default 127.0.0.1) |
Configure an API key for any supported provider. Users pick provider and model from a dropdown. Keys never leave the server.
NL_SQL_PROVIDER=minimax # default provider
MINIMAX_API_KEY=eyJ... # or ANTHROPIC_API_KEY / DEEPSEEK_API_KEY / etc.With no keys configured, the feature returns 503; nothing else is affected.
Changing RAP_SECRET directly logs out every user. Use RAP_SECRET_PREVIOUS for zero-downtime rotation:
- Generate a new secret, move the current
RAP_SECRETvalue intoRAP_SECRET_PREVIOUS, setRAP_SECRETto the new value, and restart. - Wait at least
TOKEN_LIFETIME_SECONDS(default 24h) for all old tokens to expire. - Remove
RAP_SECRET_PREVIOUSand restart.
If the old secret is suspected compromised (vs. routine rotation), skip the wait in step 2 and act immediately.
| Doc | Contents |
|---|---|
| Getting started | Install, configure, run, create your first admin |
| Architecture | Components, data model, ETL pipeline, roles, API surface, full config reference |
| NL-to-SQL | How it works, provider registry, adding a provider |
| Testing | Running the suite, synthetic dataset |
| WeChat auto-sync | Daily background sync for official-account metrics |
| Creator-portal collector | Playwright automation for XHS/Zhihu/Pugongying creator-backend exports |
| Dependency maintenance | Upgrade cadence, critical packages |
| Project comparison | Honest pros/cons vs. scraper-based and agent-workflow alternatives |
In progress:
- Video & social platform expansion (WeChat Video Channels, Douyin, Weibo, Bilibili) — tracking for stable official API/export availability on each platform before building the connector
- Xiaohongshu (XHS) Marketplace orders — e-commerce order analytics for XHS's own marketplace, alongside its existing content/KOL data
- Feishu / DingTalk push — send saved query results to team collaboration tools
Under consideration (join the Discussion):
- Multi-step NL-to-SQL with agent routing for ambiguous or multi-hop questions
- Data warehouse layering (ODS → DWD → DIM → ADS)
- Usage visualization on the admin dashboard
Issues and PRs are welcome. See CONTRIBUTING.md for:
- Local dev environment setup
- Code style and commit conventions
- Adding a platform connector or NL-to-SQL provider
- The PR checklist
- Questions & discussion — GitHub Discussions
- Bugs & feature requests — GitHub Issues
- Security vulnerabilities — see SECURITY.md; please don't open a public issue
OmniPanel is licensed under the GNU Affero General Public License v3.0 (AGPL-3.0).
You may freely use, modify, and self-host OmniPanel. Distributing a modified version, or running it as a network service, requires making the source code available under the same license.