🚀 RetailFlow: Enterprise Data Engineering Platform

🔗 Live Interactive Dashboard

https://chidex-coder.github.io/Retail-Data-Engineering-Pipeline/

_{Every filter re-aggregates the full fact table in the browser. The predictive section is model output, scored against its baselines. Shown in light and dark.}

---

📖 Project Overview

RetailFlow is a production-inspired Enterprise Data Engineering Platform that simulates the analytics infrastructure of a Fortune 500 retail company.

The platform generates millions of messy synthetic retail transactions, processes them using PySpark, transforms them into a Medallion Data Lakehouse Architecture (Bronze → Silver → Gold), models the data into a Star Schema, orchestrates the pipeline using Apache Airflow, validates data quality with Pytest, and delivers executive business insights through an interactive Plotly Dashboard.

This project demonstrates many of the technologies and engineering practices used by modern Data Engineers in production environments.

---

🏗️ Solution Architecture

                  +----------------+
                  | Synthetic Data |
                  | Generator       |
                  +--------+-------+
                           |
                           v
                 Bronze Data Lake
                  Raw CSV /Parquet
                           |
                           v
               PySpark Cleaning Jobs
     Schema Drift • Nulls • Dedup • Validation
                           |
                           v
                 Silver Data Lake
               Standardized Parquet
                           |
                           v
              Star Schema Transformation
                           |
        +------------------+------------------+
        |                  |                  |
        v                  v                  v
  DimCustomer        DimProduct         DimDate
        \                 |                 /
         \                |                /
          +---------------+---------------+
                          |
                          v
                    FactSales Table
                          |
                          v
                  Gold Analytics Layer
                          |
            +-------------+-------------+
            |                           |
            v                           v
      Plotly Dashboard           Pytest Validation
            |                           |
            +-------------+-------------+
                          |
                          v
                 Apache Airflow DAG
                          |
                          v
                    GitHub Actions CI

---

⚙️ Technology Stack

Category	Technology
Language	Python
Data Processing	Apache Spark (PySpark)
Workflow Orchestration	Apache Airflow
Storage	CSV, Parquet
Data Architecture	Medallion Data Lakehouse
Data Warehouse	Star Schema
Containerization	Docker & Docker Compose
CI/CD	GitHub Actions
Testing	Pytest
Machine Learning	scikit-learn (gradient boosting, ridge, K-means)
Dashboard	Plotly.js, embedded in a self-contained HTML page
Live app	Dash (Plotly) served by gunicorn on Fly.io · Streamlit
Notebook	Jupyter, executed and exported to GitHub Pages
Version Control	Git & GitHub

---

✨ Features

• Generate 1.2–5 million synthetic retail transactions
• Simulate real-world dirty data
• Handle schema drift automatically
• Clean and standardize data with PySpark
• Bronze → Silver → Gold Medallion Architecture
• Star Schema Data Warehouse
• Executive dashboard with browser-side cross-filtering across 8 dimensions
• Self-contained, offline-capable HTML report (light and dark themes)
• Live Dash app serving the same report from pandas, sharing one stylesheet
• Predictive layer: fulfilment risk, repeat purchase, revenue forecast, segmentation
• Every model scored against an explicit baseline, published with the result
• Executed analytics notebook, one section per question, linked from every front-end
• Automated Apache Airflow Pipeline
• Dockerized Deployment
• Automated Data Quality Testing
• GitHub Actions CI/CD Pipeline

---

📂 Project Structure

Retail-Data-Engineering-Pipeline/
│
├── generator/                    # Faker-based synthetic data generator
│   ├── run.py                    # entry point (--scale controls volume)
│   ├── config.yaml               # output paths and formats
│   ├── customers.py · products.py · orders.py
│   ├── behaviour.py              # latent business model the ML must recover
│   └── dirty_data.py             # injects the realistic quality issues
│
├── spark/
│   ├── bronze_to_silver.py       # cleaning and standardisation
│   ├── silver_to_gold.py         # star schema build
│   ├── gold_analytics.py         # Gold → dashboard payload
│   └── dashboard_builder.py      # HTML assembly and column encoding
│
├── dashboards/
│   ├── executive_dashboard.html  # generated report
│   └── assets/
│       ├── app.css               # design tokens, cards, filter bar
│       └── app.js                # client-side aggregation and charts
│
├── docs/
│   ├── index.html                # GitHub Pages copy of the dashboard
│   └── notebook.html             # rendered notebook, served on Pages
│
├── app/                          # Dash app (live version)
│   ├── main.py                   # layout + callbacks
│   ├── figures.py                # Plotly figures, mirrors app.js
│   ├── components.py             # KPI cards, filter bar
│   ├── data.py                   # loads the Gold parquet, aggregates
│   └── theme.py                  # reads colour tokens out of app.css
│
├── notebooks/
│   ├── build_notebook.py         # assembles + executes the notebook, exports HTML
│   └── retailflow_analytics.ipynb
│
├── streamlit_app.py              # Streamlit front-end (Dashboard · Notebook tabs)
├── .streamlit/config.toml        # pins the light theme
│
├── ml/                           # predictive layer
│   ├── features.py               # leakage-free feature engineering
│   └── train.py                  # trains, backtests, writes data/ml/
│
├── shared/
│   └── chart_theme.json          # chart geometry, read by both front-ends
│
├── data/
│   ├── raw/                      # Bronze
│   ├── silver/
│   └── gold/                     # fact_sales + dim_customer/product/date
│
├── airflow/dags/
│   └── retail_pipeline_dag.py
│
├── tests/
│   ├── test_gold_data.py
│   └── test_ml.py                # signal + model-quality assertions
│
├── .github/workflows/ci.yml
├── docker-compose.yml
├── Dockerfile                    # Airflow image
├── Dockerfile.dash               # Dash app image (no PySpark)
├── fly.toml                      # Fly.io deployment
└── requirements.txt

---

🛠️ Data Generation

The synthetic data generator uses Faker to create between 1.2 and 5 million retail transactions with configurable volumes.

It intentionally introduces realistic data quality issues including:

• Missing values
• Duplicate records
• Mixed date formats
• Invalid currencies
• Negative prices
• HTML tags
• Emoji characters
• Schema drift
• Incorrect data types
• Random formatting inconsistencies

This closely mimics production datasets encountered by enterprise Data Engineers.

---

🥉 Bronze Layer

Stores raw transactional data exactly as received.

Format

• CSV
• Parquet

No transformations are performed at this stage.

---

🥈 Silver Layer

PySpark cleans and standardizes the data by performing:

• Schema validation
• Data type conversion
• Null handling
• Deduplication
• Currency normalization
• Date standardization
• Invalid record filtering

Outputs are written as partitioned Parquet files for efficient analytical queries.

---

🥇 Gold Layer

The Gold Layer builds an analytical Star Schema consisting of:

• FactSales
• DimCustomer
• DimProduct
• DimDate

Business metrics surfaced from this layer:

• Revenue, cost and gross profit per order line
• Average order value and average profit margin
• Order counts by fulfilment status and payment method
• Revenue by brand, category, city and customer demographic
• Monthly revenue, profit, quantity and order-volume trends

---

⚡ Spark Optimizations

The ETL pipeline uses several Spark optimization techniques:

• Broadcast Joins
• Repartitioning
• Predicate Pushdown
• Partitioned Parquet Storage
• Lazy Evaluation
• Column Pruning

These techniques improve scalability and execution performance.

---

📊 Executive Dashboard

The Gold layer feeds a single self-contained HTML report — no server, no build step, no external requests. Open the file and it works offline.

View the live dashboard →

What's on it

8 KPI cards — Revenue, Gross Profit, Orders, Customers, Products Sold, Avg Order Value, Avg Profit Margin, Cities Served.

9 visuals, in three rows:

Row	Visuals
1	Revenue Trend · Top Products by Brand · Revenue by Category
2	Revenue by City · Revenue by Gender · Payment Methods
3	Order Status · Top Customers · Monthly Orders

Interactivity

Every filter re-aggregates the entire fact table in the browser. The Gold rows are dictionary-encoded into base64 typed arrays and embedded in the page, so one filter change recomputes all eight KPIs and all nine charts in roughly 70 ms — no server round trip, no pre-baked trace toggling.

• Filters — Year, Month, Category, Brand, Payment Method, Gender, Order Status, plus a type-ahead City search
• Metric switch — Revenue / Profit / Quantity, repainting every value-based chart and rewriting its title
• Themes — light and dark, each a separately chosen palette rather than an inverted one
• Export — the current selection downloads as CSV; Plotly's PNG export is kept, its logo and lasso/box-select tools removed

Design decisions

• Bars, not pies. Length is easier to compare than angle, so payment methods and order status are horizontal bars rather than donuts.
• Colour encodes the measure, not decoration — Revenue green, Profit blue, Quantity orange, order counts purple. Every colour was checked for contrast against its surface and for colour-vision separation.
• Order Status uses a reserved status palette (Delivered green through Cancelled red) with values always labelled, so colour never carries meaning on its own.
• No legends. Each chart shows one series at a time and the metric switch says which.
• Real date axes. Trend charts plot actual dates, so Jan 2024 reads as a month instead of the integer 202401.
• Left joins, not filters. An order line with an unresolved brand or payment method still counts toward revenue and only drops out of that one breakdown — dropping those rows outright would silently delete a fifth of the revenue. The unresolved counts are printed in the dashboard footer.

Data standardisation

spark/gold_analytics.py normalises dimension attributes before they reach a chart: HTML fragments (<b>Sony</b>) and emoji (IKEA📱💻) are stripped, and placeholder values (NULL, Unknown, -, N/A) become genuine nulls instead of showing up as categories.

Generating it

python spark/gold_analytics.py

Writes dashboards/executive_dashboard.html and docs/index.html (the GitHub Pages copy). The Airflow DAG runs this as its run_analytics task.

---

🎈 Streamlit App

The third front-end, on the same app/ modules as Dash — app/data.py aggregates, app/figures.py draws — with Streamlit supplying the page, sidebar filters and per-filter-set caching. Two tabs: Dashboard and Notebook.

pip install -r app/requirements-streamlit.txt
streamlit run streamlit_app.py            # http://localhost:8501

.streamlit/config.toml pins the theme to the dashboard's light tokens so the cards render on the surface they were designed for.

---

🤖 Predictive Analytics

Three supervised models and one segmentation, trained on the Gold layer by python -m ml.train. Every one is scored against an explicit baseline, and the baseline ships next to the result — on the dashboard and in data/ml/metrics.json.

Model	Task	Result	Baseline
Fulfilment risk	P(order returned or cancelled)	ROC AUC 0.634 · PR-AUC 0.391	chance 0.500 · base rate 0.289
Repeat purchase	P(customer orders again within 90 days)	ROC AUC 0.965	recency-only rule 0.947
Revenue forecast	monthly revenue, 3 months ahead	MAPE 12.9%	naive 20.9% · seasonal naive 35.1%
Segmentation	K-means on log RFM	4 segments over 4,100 customers	unsupervised

The generator had to change first

The original generator drew every order field independently:

"Status":        random.choice(["Pending", "Shipped", ...]),
"PaymentMethod": random.choice([...]),
"OrderDate":     fake.date_between(start_date='-3y', ...),

Independent uniform draws contain nothing to learn. Measured on that data, order status was uniform (chi-square p = 0.505) and gradient boosting scored 20.05% against a 20.00% baseline — no skill, because there was no signal. A model reporting high accuracy on it would have been fitting noise.

So generator/behaviour.py now carries a latent business model, and the generator samples from it:

• Customer segments — VIP, loyal, occasional and bargain accounts with different order rates, basket sizes, category affinities and churn hazards
• Seasonality — a Q4 run-up and a January trough, with compounding annual growth and a weekend uplift
• Price structure — category price bands and margins, with quantity responding to price
• Fulfilment behaviour — returns driven by category (Clothing high, Books low), cancellations by payment method (bank transfer high, wallets low), both rising with order value
• UK geography — real cities carrying population-weighted volume

None of it is written to the output files. The models have to rediscover it, which is the point.

No leakage

• Customer-history features are built with expanding windows and a shift(), so an order's own outcome never informs its own features. A first order carries no history and is flagged as such.
• The fulfilment split is chronological, not random — a random split would leak later behaviour of the same customers into training.
• The repeat-purchase label is measured strictly after the feature cutoff.
• The forecast is backtested with rolling origin, refitting on only the past at each step, and its regularisation strength is tuned inside each fold. Tuning it against the backtest would leak the very error being reported.
• Revenue = Quantity × SellingPrice is a deterministic identity in this schema, so no model predicts revenue from its own components.

tests/test_ml.py asserts both halves: that the data still carries the signal, and that each model still beats its baseline. Flatten the generator back to uniform draws and those tests fail loudly rather than the models quietly degrading.

Running it

python spark/gold_analytics.py --table-only   # publish the analytics table
python -m ml.train                            # train, evaluate, write artefacts
python spark/gold_analytics.py                # rebuild the dashboard with the models

Artefacts land in data/ml/ (metrics, forecast, risk tables, segments). Both dashboards render the predictive section when they are present and simply omit it when they are not.

---

📓 Analytics Notebook

The whole analysis as a readable, executed notebook: notebooks/retailflow_analytics.ipynb — or the rendered copy on GitHub Pages: notebook.html.

One section per dashboard visual, each with the question it answers, the pandas that answers it, the resulting table and the chart:

KPIs	the eight executive numbers
Q1–Q9	revenue trend · brands · categories · cities · gender · payment methods · order status · top customers · monthly orders
Q10–Q12	the signal checks that justify a predictive layer — return rate by category, cancellation rate by payment method, seasonality index
Models 1–4	fulfilment risk, repeat purchase, revenue forecast and segmentation, each trained by the code in ml/train.py and scored against its baseline, with prediction tables
Dashboards	the three front-ends, with the static report embedded

Figures are rendered to PNG so they display on GitHub without a kernel. Every chart reads its colours from dashboards/assets/app.css and its geometry from shared/chart_theme.json — the same sources the dashboards use.

All three front-ends carry a Notebook tab that opens it.

Rebuilding it

pip install -r notebooks/requirements.txt
python notebooks/build_notebook.py     # assembles, executes, exports docs/notebook.html

---

🖥️ Live Dash App

The same report, served by Python instead of precomputed into a file — a Dash app in app/, deployed to Fly.io.

Why both? A static file can only answer questions it was built to answer. The eight filters combine into 1.6 billion possible selections, so precomputing them all is impossible; the static build embeds the fact rows and aggregates them in the browser instead. The Dash app aggregates the same rows in pandas, per request. Two answers to the same constraint, and the contrast is the point:

	Static report	Dash app
Aggregation	JavaScript, in the browser	pandas, on the server
Delivery	one file, opens offline	a running process
Hosting	GitHub Pages	Fly.io
Filter response	~70 ms	~26 ms compute + round trip
Always available	yes	yes (machine kept warm)

Shared by construction

The two front-ends look identical because they read the same sources, not because anyone kept them in sync by hand:

• dashboards/assets/app.css — Dash serves this folder directly, so both use one stylesheet
• shared/chart_theme.json — bar radius, gaps, margins, tick formats
• Colours — declared once as CSS custom properties; app.js reads them via getComputedStyle, app/theme.py parses the same declarations
• data/gold/analytics_facts.parquet — the cleaned fact table both consume (1.6 MB, loads in 9 ms)

The only deliberate duplication is trace assembly: ~150 lines of app/figures.py mirroring app.js, which no amount of sharing can avoid across two languages.

Running it

pip install -r app/requirements.txt
python run_dash.py            # http://localhost:8050

The Docker image excludes PySpark entirely — the app reads the pre-built parquet, so it needs no JVM at runtime.

docker build -f Dockerfile.dash -t retailflow-dash .
flyctl deploy

---

🔄 Apache Airflow

The entire ETL pipeline is fully automated using Apache Airflow.

Pipeline:

Generate Data
      ↓
Bronze → Silver
      ↓
Silver → Gold
      ↓
Generate Dashboard
      ↓
Run Pytest

Airflow provides:

• Scheduling
• Monitoring
• Retry Logic
• Pipeline Dependency Management

---

✅ Data Quality

Automated testing is implemented using Pytest.

Current validation includes:

• Null checks
• Positive revenue validation
• Primary key uniqueness
• Referential integrity
• Duplicate detection

The pipeline automatically fails if validation tests do not pass.

---

🚀 CI/CD

GitHub Actions automatically executes on every push.

Pipeline steps include:

• Install dependencies
• Validate project
• Execute PySpark jobs
• Run Pytest suite
• Publish build status

---

💻 Running Locally

Clone Repository

git clone https://github.com/chidex-coder/Retail-Data-Engineering-Pipeline.git

Enter Project

cd Retail-Data-Engineering-Pipeline

Create Virtual Environment

python -m venv venv

Windows

venv\Scripts\activate

macOS/Linux

source venv/bin/activate

Install Dependencies

pip install -r requirements.txt

Run the Pipeline Directly

Runs the whole pipeline without Docker (requires a JRE for PySpark):

python generator/run.py --scale 0.01
python spark/bronze_to_silver.py
python spark/silver_to_gold.py
python spark/gold_analytics.py --table-only
python -m ml.train
python spark/gold_analytics.py
python -m pytest tests/ -v

Then open dashboards/executive_dashboard.html in a browser.

Run the Live Dash App

pip install -r app/requirements.txt
python run_dash.py

Serves on http://localhost:8050. Needs data/gold/analytics_facts.parquet, which the pipeline step above produces.

Start Airflow

docker compose up --build

Open Airflow:

http://localhost:8082

Default Credentials:

Username: admin

Password: admin

Trigger:

retail_flow_pipeline

---

📈 Skills Demonstrated

• Python
• SQL
• PySpark
• Apache Spark
• Apache Airflow
• Docker
• Docker Compose
• GitHub Actions
• Plotly
• ETL
• ELT
• Data Engineering
• Data Warehousing
• Star Schema
• Medallion Architecture
• Data Lakehouse
• Data Quality
• Data Validation
• Automation
• CI/CD

---

🏆 Project Outcomes

✔ Generated over 5 million synthetic retail transactions

✔ Built an end-to-end enterprise ETL pipeline

✔ Implemented Bronze → Silver → Gold architecture

✔ Automated data quality validation

✔ Delivered executive analytics dashboard

✔ Containerized the entire platform with Docker

✔ Automated orchestration using Apache Airflow

✔ Integrated continuous integration with GitHub Actions

✔ Demonstrated production-ready Data Engineering workflows

---

🔮 Future Improvements

• Kafka Streaming
• Delta Lake
• dbt Transformations
• Great Expectations
• AWS S3 Data Lake
• Snowflake Integration
• Terraform Infrastructure
• Kubernetes Deployment

---

👨‍💻 Author

Chiagoziem Cyriacus Ugoh

Full Stack Data Professional | Data Engineer | Data Analyst | Python Developer

GitHub: https://github.com/chidex-coder

---

⭐ If you found this project interesting, consider giving it a Star!