Hybrid Reinforcement Learning (RL) with Stochastic Network Optimisation (SNO) for Restaurant Inventory Management

Problem Statement & Motivation
Research Objectives
Hybrid RL-SNO Framework Architecture
Project Structure
Datasets
MDP Environment — Complete Design
RL Agents — Complete Implementation
Classical Baselines
Training Protocol — Step by Step
Evaluation Protocol
Experimental Scenarios
Scenario 1 Results — Generic Restaurant
Scenario 2 Results — Five Bangladeshi Restaurants
Cross-Restaurant Transfer Test
Key Findings, Reasons & Insights
Research Contributions
Limitations & Future Work
How to Run
Requirements

1. Problem Statement & Motivation

1.1 The Core Business Problem

Every restaurant faces a daily, unavoidable inventory dilemma. Order too much and perishable ingredients spoil — Mutton at BDT 900/kg with a two-day shelf life represents thousands of taka in waste every day the order is miscalibrated. Order too little and the kitchen runs out mid-service — customers who ordered Kacchi Biryani are turned away, revenue is lost, and reputation suffers.

This dual failure mode costs the global restaurant industry over USD 160 billion per year. The problem is structurally hard because it is not static: the right quantity to order today is different from the right quantity to order next Friday (when demand is 60% higher), completely different during Ramadan (when iftar dishes surge 80% while lunch items collapse 50%), and radically different on Eid-ul-Fitr (when demand reaches 2.5× normal).

1.2 Why Classical Methods Fail

Classical inventory theory has produced elegant solutions — but they are solutions to a different problem. Economic Order Quantity (Wilson, 1913) minimises ordering and holding costs under static, known demand. The Newsvendor model handles stochastic demand but over a single period with no carry-over. Neither can capture:

Weekly cyclicality: Friday demand at Bangladeshi restaurants is 60–70% above Monday
Cultural seasonality: Ramadan restructures the entire demand distribution for 30 days, with Eid-ul-Fitr producing 2.5× normal demand
Network coupling: a decision to order more Mutton simultaneously affects Kacchi Biryani, Mutton Rezala, and Mixed Platter — these dishes share ingredients and cannot be managed independently
Perishability dynamics: an ingredient ordered today may expire before use — the timing of orders relative to shelf life is as important as the quantity

1.3 The Research Gap

RL has produced superhuman agents for Go, Dota, and protein folding — domains with complex, long-horizon, stochastic decision-making. Restaurant inventory has all of these characteristics. Yet very few RL frameworks have been validated on real, culturally specific restaurant data with proper train/test temporal splits that prove generalisation to future demand.

This project fills that gap. We design, implement, and validate a hybrid RL-SNO framework on operational data from five real Bangladeshi restaurants — five cuisines, five Dhaka districts, five completely different operational schemas.

2. Research Objectives

Design a unified hybrid RL-SNO framework jointly modelling the ingredient-meal bipartite network and learning ordering policies through deep RL to solve the Core Business Problem with dual failure.
Implement four RL agents from mathematical first principles in pure NumPy (Q-Learning, DQN, Double DQN, PPO) — no PyTorch or TensorFlow.
Benchmark against four classical baselines under identical conditions with dual profit-margin metrics.
Validate with train/test temporal splits (80/20 chronological) — evaluate only on held-out future data.
Demonstrate zero-shot cross-restaurant transfer — train on four restaurants, evaluate on the fifth.
Produce a reproducible empirical study with 29 unit tests, multi-seed evaluation, and honest reporting of negative results alongside positive ones.

3. Hybrid RL-SNO Framework Architecture

3.1 System Overview

╔═══════════════════════════════════════════════════════════════════════╗
║             STOCHASTIC NETWORK OPTIMISATION (SNO) LAYER               ║
║                                                                       ║
║  SUPPLY NODES              EDGES A[i,j]              DEMAND NODES     ║
║  (ingredients)         units of i per serving j      (dishes)         ║
║                                                                       ║
║  Rice      (10d SL) ──── 3.0 ─────────────────────► Kacchi Biryani    ║
║  BDT 80/u           ──── 3.0 ─────────────────────► Chicken Tehari    ║
║                                                                       ║
║  Mutton    (2d SL)  ──── 2.0 ─────────────────────► Kacchi Biryani    ║
║  BDT 900/u          ──── 2.0 ─────────────────────► Mutton Rezala     ║
║                     ──── 1.0 ─────────────────────► Mixed Platter     ║
║                                                                       ║
║  Chicken   (2d SL)  ──── 2.0 ─────────────────────► Chicken Tehari    ║
║  BDT 350/u          ──── 3.0 ─────────────────────► Chicken Roast     ║
║                     ──── 1.0 ─────────────────────► Mixed Platter     ║
║                                                                       ║
║  Onion, Garam Masala ──── shared across 4-5 meals ──────────────────► ║
╚═══════════════════════════════════════════════════════════════════════╝
                              ↕  State observation s_t (17-dim)
╔═══════════════════════════════════════════════════════════════════════╗
║                   REINFORCEMENT LEARNING LAYER                        ║
║                                                                       ║
║  s_t ∈ ℝ¹⁷ ──► Agent (Q / DQN / DDQN / PPO) ──► a_t ∈ {0..242}        ║
║                                                                       ║
║  r_t = (Revenue − OrderCost − WasteCost                               ║
║         − StockoutPenalty − HoldingCost) / reward_scale               ║
║                                                                       ║
║  Objective: maximise Σ γᵗ rₜ over a 30-day episode                     ║
╚═══════════════════════════════════════════════════════════════════════╝

3.2 The SNO Layer — Bipartite Network

The supply chain is a directed bipartite graph G = (S ∪ D, E, A) where:

S = supply nodes (ingredients): unit cost, shelf life, FIFO inventory batches
D = demand nodes (meals): price, stochastic daily demand
A ∈ ℝ⁵ˣ⁵ = ingredient-meal matrix where A[i,j] = units of ingredient i per serving of meal j

Kachchi Bhai — A matrix:

	Kacchi Biryani	Chicken Tehari	Chicken Roast	Mutton Rezala	Mixed Platter
Kataribhog Rice	3.0	3.0	0	0	2.0
Mutton	2.0	0	0	2.0	1.0
Chicken	0	2.0	3.0	0	1.0
Onion	1.0	0.5	0.5	0.5	0.5
Garam Masala	1.0	0.5	1.0	1.0	0.5

Every ordering decision must satisfy all ingredient-meal coupling constraints simultaneously. This is the SNO layer operating at each step.

3.3 Information Flow — One Complete Step

Step t:
  1. Agent observes s_t = [inv_norm | age_ratios | day_enc | demand_ema]
  2. Agent selects a_t ∈ {0,...,242} → order_qty[5 ingredients]
  3. ORDER: record (arrival_day = t + lead_time, order_qty) in pipeline
  4. RECEIVE: deliver any pipeline orders due at step t → append new batches (age=0)
  5. SPOIL: age all batches +1; discard batches with age ≥ shelf_life
     → waste_cost = Σᵢ spoiled[i] × unit_cost[i] × waste_multiplier
  6. DEMAND: sample demand[j] ~ N(μ_dow[j], σ_dow[j]) × calendar_overlay(t)
  7. FULFIL: serve meals in descending price order via inventory constraints
     → served[j] = min(feasible_from_inventory, demand[j])
     → stockout[j] = demand[j] − served[j]
  8. REWARD:
     revenue          = Σⱼ served[j] × price[j]
     stockout_penalty = Σⱼ stockout[j] × price[j] × penalty_factor
     holding_cost     = Σᵢ inventory[i] × holding_rate
     raw_reward       = revenue − order_cost − waste_cost − stockout_penalty − holding_cost
     reward           = raw_reward / reward_scale
  9. UPDATE: demand_ema[j] = 0.3 × demand[j] + 0.7 × ema[j]
 10. Transition to s_{t+1}

3.4 Key Design Innovations

Feature	Description	Motivation
FIFO batch aging	Each ingredient holds a deque of `{qty, age}` batches; oldest expires first	Physically correct perishability model
Lead-time replenishment	Orders arrive after `lead_time` days via pipeline list	Forces anticipatory ordering
Ingredient-specific action levels	Fine=[0,25,50]; Medium=[0,50,100]; Coarse=[0,75,150]	Precise control for perishables; coarse control for stable items
Calendar-offset episode starts	`start_day ~ U[0, split_day)`	Episodes run into Ramadan during training
Variable episode length	`ep_len ~ U[20, ep_len_train]` during training	Prevents horizon-specific policy overfitting
Reward normalisation	`reward / reward_scale` (BDT: 1.2–2.3; USD: 1.0)	Equalises gradient magnitude across currencies
Train/test temporal split	80% train / 20% test chronological	Validates true future generalisation
Dual PM metrics	Financial PM (cash) + RL-Adjusted PM (reward-aligned)	Reveals degenerate conservative policies

4. Project Structure

project/
│
├── main.py                     # Master pipeline runner — 5 steps, CLI flags
├── config.py                   # All hyperparameters, 5-restaurant dicts, action spaces
├── restaurant_env.py           # Complete MDP: FIFO, lead time, empirical demand, dual PM
├── rl_agents.py                # NumpyMLP + Q-Learning + DQN + DDQN + PPO from scratch
├── classical_baselines.py      # EOQ + Newsvendor + SMA+Reorder + Seasonal Reorder
├── training_loop.py            # Train/eval engine, LR decay, transfer evaluation
├── data_loader.py              # Scenario 1: 8 dataset loaders with graceful fallback
├── eda_scenario1.py            # 8 EDA figures from public datasets
├── eda_scenario2.py            # 8 EDA figures from restaurant datasets
├── scenario1_generic.py        # Scenario 1 runner (train/test split)
├── scenario2_kacchi_bhai.py    # Scenario 2: all 5 restaurants + transfer evaluation
├── visualisations.py           # 14 RL figures: curves, KPI, bipartite networks, radars
├── results_summary.py          # Dynamic tables, 7 findings, contributions
├── requirements.txt
│
├── tests/
│   └── test_core.py            # 29 unit tests, 10 test classes
│
└── data/
    ├── meal_info.csv                                    # 51 items, 14 categories
    ├── Grocery_Inventory_and_Sales_Dataset.csv         # 990 SKUs
    ├── fulfilment_center_info.csv                      # 77 centres
    ├── Food_Ingredients_and_Recipe_Dataset_...csv      # 13,501 recipes
    ├── kacchi_bhai_daily_orders.csv                    # 182 rows (wide daily)
    ├── star_kabab_daily_orders.csv + _wide.csv         # 912 rows
    ├── fakruddin_daily_orders.csv + _wide.csv          # 1,095 rows
    ├── madchef_daily_orders.csv + _wide.csv            # 548 rows (120,607 item records)
    ├── bismillah_daily_orders.csv + _wide.csv + _weekly_summary.csv  # 730 rows
    ├── Instacart_2017/          ← download and place here
    ├── Open_Food_Facts/         ← download and place here
    ├── UCI_Online_Retail/       ← download and place here
    └── Zomato_Restaurants/      ← download and place here

5. Datasets

5.1 Scenario 1 — Eight Public Reference Datasets

#	Dataset	Size	Role
1	`meal_info.csv`	51 items, 14 categories, 4 cuisines	Menu structure calibration
2	`Grocery_Inventory_and_Sales_Dataset.csv`	990 SKUs	Unit cost calibration (Grains $2.75, Seafood $9.50)
3	`fulfilment_center_info.csv`	77 centres	Supply capacity; mean reorder level 51.2
4	`Food_Ingredients_and_Recipe_Dataset.csv`	13,501 recipes	Ingredient co-occurrence; top: salt (4,000+ recipes)
5	Instacart 2017	3,421,083 orders	Day-of-week demand profile (peak: Monday)
6	Open Food Facts	2.7M+ products	Shelf-life proxies per category
7	Zomato Restaurants	9,551 restaurants	Cuisine demand weights; top-3: North Indian, Chinese, Fast Food
8	UCI Online Retail	541,909 transactions	Reorder quantity benchmarks

Datasets 5–8 use graceful fallback if absent (pipeline still runs on datasets 1–4).

5.2 Scenario 2 — Five Bangladeshi Restaurants

Operational data collected under private data-sharing agreements for educational and research purposes.

Schema Diversity

Restaurant	Schema	Rows	Date Format
Kachchi Bhai	Wide daily (1 row/day, 1 col/meal)	182	YYYY-MM-DD
Star Kabab	POS long-format (1 row/meal/day)	912	DD/MM/YYYY
Fakruddin	Recipe-based (DayType, PortionsSold, GrossProfit)	1,095	ISO date
Madchef	Item-level order log (timestamp, payment, table)	548 wide / 120,607 raw	timestamp
Bismillah	Weekly aggregate + daily (Tk abbreviation)	730	DD-Mon-YYYY

Restaurant Profiles

Restaurant	Location	Cuisine	Period	Days	Train	Test	Lead Time
Kachchi Bhai	Dhanmondi	Biryani/Mughlai	Oct 2024–Mar 2025	182	0–145	145–182 (Ramadan!)	1d
Star Kabab	Old Dhaka	Kebab/Grill	Jan 2021–Jun 2023	912	0–729	729–912	1d
Fakruddin	Gulshan	Traditional Mughlai	Jan 2022–Dec 2024	1,095	0–876	876–1095	2d
Madchef	Banani	Modern Fast Food	Jul 2023–Dec 2024	548	0–438	438–548	1d
Bismillah Hotel	Mirpur-10	Local Thali	Jan 2021–Dec 2022	730	0–584	584–730	1d

Ingredient Details Per Restaurant

Kachchi Bhai:

Kataribhog Rice: BDT 80/u, 10d SL | Mutton: BDT 900/u, 2d SL | Chicken: BDT 350/u, 2d SL
Onion: BDT 40/u, 5d SL | Garam Masala: BDT 120/u, 90d SL

Star Kabab:

Beef: BDT 750/u, 2d SL | Chicken: BDT 320/u, 2d SL | Onion: BDT 35/u, 5d SL
Spices: BDT 200/u, 60d SL | Bread: BDT 15/u, 2d SL

Fakruddin (2-day lead time, most perishable ingredients):

Basmati Rice: BDT 120/u, 14d SL | Mutton: BDT 950/u, 2d SL
Hilsha Fish: BDT 600/u, 1d SL (most perishable in entire study)
Lentils: BDT 80/u, 30d SL | Spice Blend: BDT 180/u, 90d SL

Madchef (all primary ingredients 2-day SL):

Chicken Breast: BDT 320/u, 2d SL | Buns: BDT 25/u, 2d SL | Cheese: BDT 180/u, 7d SL
Lettuce: BDT 30/u, 4d SL | Sauce: BDT 90/u, 30d SL

Bismillah Hotel (low-margin — prices BDT 70–160):

Rice: BDT 55/u, 30d SL | Lentils: BDT 90/u, 14d SL | Vegetables: BDT 30/u, 3d SL
Mustard Oil: BDT 160/u, 60d SL | Dry Fish: BDT 200/u, 30d SL

6. MDP Environment — Complete Design

File: restaurant_env.py

6.1 Initialisation

RestaurantInventoryEnv(
    scenario_cfg,         # restaurant config dict (config.py)
    empirical_csv=None,   # path to CSV → empirical demand; None → parametric
    seed=42,
    mode='train',         # 'train' | 'test' | 'all'
    init_inv_noise=0.20   # ±20% noise on initial inventory at each reset
)

On init: builds the A matrix, constructs 243 joint actions via itertools.product, computes split_day = int(total_days × 0.80), and if a CSV is given calls _load_empirical_demand() which reads the CSV, splits at split_n = int(len(df) × 0.80), and builds per-DOW (μ, σ) distributions from the training rows.

6.2 Reset

def reset():
    # Episode length (variable in training, fixed in evaluation)
    if mode == 'train':
        ep_length = rng.integers(20, episode_length_train + 1)
    else:
        ep_length = 30

    # Calendar start (chronological split enforcement)
    if mode == 'train':
        start_day = rng.integers(0, split_day)      # can run past split_day into Ramadan
    elif mode == 'test':
        start_day = rng.integers(split_day, total_days − ep_length)
    else:
        start_day = rng.integers(0, total_days − ep_length)

    # Initial inventory with noise (generalisation best practice)
    inventory = clip(init_inv × rng.uniform(0.80, 1.20), 1, MAX_INVENTORY=100)
    _ep_initial_inv_total = inventory.sum()  # for correct waste denominator

    # FIFO deques — one fresh batch per ingredient
    inv_batches[i] = deque([{'qty': inventory[i], 'age': 0.0}])
    pipeline = []
    demand_ema = base_demand.copy()
    return _get_state()

6.3 State Space — 17 Dimensions

Dims	Content	Formula	Range
0–4	Inventory norm	`inventory[i] / 100`	[0, 1]
5–9	FIFO age ratio	`mean_batch_age[i] / shelf_life[i]`	[0, ∞)
10–11	Circular DOW	`sin(2π×dow/7)`, `cos(2π×dow/7)`	[−1, +1]
12–16	Demand EMA	`ema[j] / max(base_demand)`	[0, ∞)

Circular day encoding: Friday (dow=4) and Saturday (dow=5) are geometrically close in (sin, cos) space, so the agent can generalise from "Friday surge" to "Saturday also high" without separately learning each. One-hot encoding treats every day as equally distant from every other.

Age ratio in state: if Mutton has age=1.5 and shelf_life=2, the ratio is 0.75 — the agent knows it must be used today or tomorrow. This drives conservative reorder on old batches and prevents the agent from blindly ordering more when existing stock is close to expiry.

Demand EMA: updated at each step as ema[j] = 0.3 × demand[j] + 0.7 × ema[j]. Provides a noisy but informative signal about recent demand levels. During Ramadan, the EMA of Kacchi Biryani demand shifts upward over ~3 days, giving the agent early warning to increase Rice and Mutton orders.

For Q-Learning: the 17-dim state is discretised to (4 bins)^5 × 7 = 7,168 states by binning inventory levels into quartiles and decoding DOW from the circular encoding. Age and EMA are discarded. This information loss is intentional and acknowledged.

6.4 Action Space — 243 Joint Combinations

# Ingredient-specific order levels (Minor-6 fix)
ORDER_LEVELS_FINE   = [0,  25,  50]   # Mutton, Chicken, Beef, Hilsha Fish, Buns
ORDER_LEVELS_MEDIUM = [0,  50, 100]   # Rice, Onion, Lentils, Cheese
ORDER_LEVELS_COARSE = [0,  75, 150]   # Cooking Oil, Spice Mix, Sauce, Mustard Oil

# Joint action space: Cartesian product of 5 ingredients × 3 levels = 3^5 = 243
combos, order_matrix = build_action_space(cfg)
# order_matrix.shape = (243, 5)
# order_matrix[0] = [0, 0, 0, 0, 0]   — do not order anything
# order_matrix[1] = [0, 0, 0, 0, 75]  — order 75 Cooking Oil only
# ...

Why ingredient-specific levels: Mutton (BDT 900/kg, 2d SL) requires fine control — the difference between ordering 25 and 50 units is enormous financially and spoilage-wise. Cooking Oil (BDT 160/u, 60d SL) is forgiving; coarse steps of 75 or 150 waste action-space resolution on an ingredient where precision doesn't matter.

6.5 FIFO Batch Aging — Exact Implementation

def _step_spoilage():
    for i in range(n_ing):
        new_deque = deque()
        total_spoiled = 0.0
        for batch in inv_batches[i]:
            batch['age'] += 1.0
            if batch['age'] >= shelf_life[i]:  # expired
                total_spoiled += batch['qty']
            else:
                new_deque.append(batch)
        inv_batches[i] = new_deque
        spoiled[i] = total_spoiled

    waste_cost = sum(spoiled[i] * unit_cost[i] * waste_cost_multiplier for i in range(n_ing))
    inventory -= spoiled
    total_units_wasted += spoiled.sum()

def _consume_fifo(ing_idx, quantity):
    remaining = quantity
    new_deque = deque()
    for batch in inv_batches[ing_idx]:
        if remaining <= 0:
            new_deque.append(batch)
        elif batch['qty'] <= remaining:
            remaining -= batch['qty']     # consume entire old batch
        else:
            batch['qty'] -= remaining     # partially consume
            remaining = 0
            new_deque.append(batch)
    inv_batches[ing_idx] = new_deque

Why FIFO matters for training: without FIFO, an agent might learn "high inventory = safe" when in fact it has a large quantity of 1-day-old Mutton about to expire. The age ratio in the state vector directly reflects the FIFO batch structure, giving the agent the information it needs to manage perishability correctly.

6.6 Demand Sampling — Empirical + Calendar Overlay

def _sample_demand(abs_day):
    dow = abs_day % 7

    # Always use training-period distributions (both modes)
    # Avoids unreliable test distributions from only 20% of rows
    # (~37 rows for Kachchi Bhai = ~5 data points per day-of-week)
    emp = emp_demand_by_dow_train  # {dow: {meal: (mu, sd)}}

    if emp is not None:
        demand[m] = max(0, Normal(emp[dow][m]))  # empirical per-DOW
    else:
        demand = _parametric_demand(abs_day, dow)  # parametric fallback

    return _apply_calendar_overlays(demand, abs_day)

def _apply_calendar_overlays(demand, abs_day):
    if ramadan_start <= abs_day < ramadan_start + 30:
        for iftar_meal in ramadan_shift['iftar_meals']:
            demand[iftar_meal] *= peak_multiplier      # 1.6–2.0×
        for other_meal:
            demand[other_meal] *= off_peak_factor      # 0.3–0.6×
    if abs_day in eid_days:
        demand *= eid_spike_multiplier                 # 1.8–3.0×
    return demand.round().clip(0)

Calendar overlay values per restaurant:

Restaurant	Friday boost	Saturday boost	Ramadan peak mult.	Ramadan off-peak	Eid spike
Kachchi Bhai	1.60×	1.40×	1.80× (iftar)	0.50×	2.50×
Star Kabab	1.70×	1.50×	2.00× (iftar)	0.30×	2.20×
Fakruddin	1.55×	1.35×	1.60× (biryani)	0.40×	3.00×
Madchef	1.45×	1.60×	1.30×	0.60×	1.80×
Bismillah	1.30×	1.20×	1.40×	0.55×	2.00×

6.7 Reward Function — Full Decomposition

raw_reward = revenue − order_cost − waste_cost − stockout_penalty − holding_cost
reward     = raw_reward / reward_scale

Component	Formula
revenue	`Σⱼ served[j] × meal_price[j]`
order_cost	`Σᵢ order_qty[i] × unit_cost[i]`
waste_cost	`Σᵢ spoiled[i] × unit_cost[i] × waste_mult` (1.4–1.8× depending on restaurant)
stockout_penalty	`Σⱼ stockout[j] × meal_price[j] × stockout_factor` (0.7–0.9×)
holding_cost	`Σᵢ inventory[i] × holding_rate` (BDT 0.03–0.06/unit)

reward_scale: Kachchi Bhai=2.0, Star Kabab=1.9, Fakruddin=1.6, Madchef=2.3, Bismillah=1.2, Generic=1.0.

Without normalisation, BDT environments produce raw daily rewards of ±50,000 BDT. Neural networks trained on these magnitudes produce Q-value targets of ±1.5M (over 30 days with γ=0.95), causing gradient instability. Scaling to ±25,000 (Kachchi Bhai ÷2.0) brings the learning problem into a more tractable range.

6.8 Dual Profit-Margin Metrics

def get_episode_metrics():
    financial_cost = order_cost + waste_cost + holding_cost

    # Financial PM: what the accountant sees — no stockout penalty in P&L
    financial_pm = (revenue − financial_cost) / revenue × 100

    # RL-Adjusted PM: reward-aligned — penalises unmet demand
    rl_adjusted_pm = (revenue − financial_cost − stockout_penalty) / revenue × 100

    # Waste % — CORRECTED denominator (v5 fix)
    # Old: wasted / max(ordered, 1)  ← inflated when agent orders near-zero
    # New: wasted / (initial_inv_at_reset + received_during_episode)
    total_available = _ep_initial_inv_total + total_units_received
    waste_pct = total_units_wasted / max(total_available, 1.0) × 100

    service_level = total_meals_served / max(total_meals_demanded, 1.0) × 100

Both PM metrics clamped to [−500%, 100%]. Waste and SL clamped to [0%, 100%].

7. RL Agents — Complete Implementation

File: rl_agents.py Principle: All four agents implemented from mathematical first principles in pure NumPy. He initialisation, Adam optimiser, cosine LR decay.

7.1 NumpyMLP — Shared Neural Backbone

Architecture:   input(17) → Dense(128, ReLU) → Dense(128, ReLU) → output(243)
Init:           W ~ N(0, sqrt(2 / fan_in))    (He / Kaiming uniform)
Optimiser:      Adam   β₁=0.9, β₂=0.999, ε=1e-8
LR schedule:    cosine  lr(t) = lr_min + 0.5×(lr₀−lr_min)×(1+cos(π×t/T))

set_lr(new_lr) updates Adam's learning rate in-place without resetting momentum, preserving accumulated gradient statistics while slowing the step size.

7.2 Q-Learning (Low-Capacity Benchmark †)

Reference: Watkins & Dayan (1992)

Q(s,a) ← Q(s,a) + α × [r + γ × max_{a'} Q(s',a') − Q(s,a)]

α=0.15  γ=0.95  ε: 1.0→0.05 (decay 0.995/episode)
Q-table: shape (7168, 243), init Uniform(−0.01, 0.01), size ~14 MB
Training episodes: 800

Acknowledged limitation: 7,168-bin state discards age ratios and demand EMA — intentional information loss. Included as a low-capacity benchmark (†), not as a competitor to neural agents on high-dimensional tasks.

7.3 DQN — Deep Q-Network

Reference: Mnih et al. (2015)

def update():
    s, a, r, s2, done = replay_buffer.sample(batch_size=64)
    q_next   = target_network.predict(s2).max(axis=1)
    q_target = r + γ × q_next × (1 − done)        # Bellman target
    targets  = online_network.predict(s)
    targets[range(64), a] = q_target               # update selected action only
    online_network.train_step(s, targets)          # MSE loss

# Every 20 episodes:
target_network.copy_weights_from(online_network)

# Hyperparameters
lr₀=0.001, lr_min=0.0001 (cosine/100 eps)
ε: 1.0→0.05 (decay 0.995/ep)
hidden=128, buffer=8000, batch=64, target_update=20, γ=0.95
Training: 600 episodes

The overestimation problem: DQN uses the same network for both selecting and evaluating the best next action. When Q-values are noisy, max_{a'} Q(s',a') selects the overestimated action and propagates the overestimate. In perishable inventory, this drives over-ordering → massive spoilage → catastrophic DQN collapse at Madchef (89% waste).

7.4 Double DQN — Decoupled Update

Reference: van Hasselt et al. (2016)

One change from DQN:

def update():
    s, a, r, s2, done = replay_buffer.sample(64)

    # DQN:    max_{a'} Q(s', a'; θ⁻)  — same biased network selects AND evaluates
    # DDQN:
    best_a   = online_network.predict(s2).argmax(axis=1)   # online selects
    q_tgt_s2 = target_network.predict(s2)                  # target evaluates
    q_sel    = q_tgt_s2[range(64), best_a]
    y = r + γ × q_sel × (1 − done)
    # rest identical to DQN

Effect: the target network acts as a second opinion on the online network's action selection. When online overestimates Q(s', "order 50 Mutton"), the target network is less likely to confirm it. This breaks the positive feedback loop responsible for DQN's waste cascade.

Empirical evidence: Star Kabab DQN = −500% FinPM, 74% waste. Star Kabab DDQN = +72.6% FinPM, 28.5% waste. Same architecture, same hyperparameters, same data — only the target computation differs.

lr₀=0.001, lr_min=0.0001 (cosine/100 eps)
ε: 1.0→0.05 (decay 0.995/ep) | hidden=128 | buffer=8000 | batch=64
Training: 600 episodes

7.5 PPO — Proximal Policy Optimisation

Reference: Schulman et al. (2017)

Actor-critic. Actor π(a|s; θ): softmax over 243 actions. Critic V(s; φ): scalar state value.

def update():
    T = len(trajectory)

    # Monte Carlo returns (full episode, low bias)
    returns = zeros(T)
    G = 0.0
    for t in reversed(range(T)):
        G = rewards[t] + γ × G
        returns[t] = G
    returns = normalise(returns)  # zero-mean, unit-std

    for epoch in range(6):  # update_epochs = 6
        # Critic: minimise MSE(V(s), return)
        critic.train_step(states, returns[:, None])

        values     = critic.predict(states).flatten()
        advantages = normalise(returns − values)

        # Actor: PPO clip + entropy bonus
        logits     = actor.predict(states)
        probs      = softmax(logits)                          # (T, 243)
        log_new    = log(probs[range(T), actions])
        ratio      = exp(log_new − log_probs_old)

        # Clip: only include gradient if ratio in [1−ε, 1+ε]
        not_clipped = ((adv ≥ 0) AND (ratio ≤ 1+0.20)) OR
                      ((adv < 0) AND (ratio ≥ 1−0.20))
        eff_adv = advantages × not_clipped

        # Score-function policy gradient (correct derivation):
        # ∇θ log π(a|s) applied via cross-entropy: one_hot(a) − π(·|s)
        pg_grad  = eff_adv[:, None] × (probs − one_hot(actions)) / T
        ent_grad = −0.01 × (log(probs) + 1) / T    # entropy bonus
        actor.train_step_with_grad(states, pg_grad + ent_grad)

    # Clear trajectory buffers
    states, actions, rewards, log_probs = [], [], [], []

# Hyperparameters
lr_actor₀=0.001, lr_actor_min=0.0001
lr_critic₀=0.003, lr_critic_min=0.0003  (cosine decay)
γ=0.95, clip_ε=0.20, update_epochs=6, entropy_coef=0.01
Training: 500 episodes

Why entropy bonus: without it, the softmax distribution collapses early (entropy → 0), producing a near-deterministic policy before the agent has explored the space sufficiently. The 0.01 coefficient penalises low-entropy policies throughout training.

PPO vs DQN timing: DQN updates every step (one replay-buffer sample per step). PPO updates once per episode (using the full trajectory). DQN makes more gradient updates per wall-clock time; PPO updates are lower-bias because they use full Monte Carlo returns.

8. Classical Baselines

File: classical_baselines.py

8.1 EOQ — Economic Order Quantity (Wilson, 1934)

Q* = sqrt(2 × D × K / h)
Reorder when inventory[i] ≤ reorder_threshold[i]

D = estimated daily demand × lead_time. Ignores demand variance, seasonality, perishability.

8.2 Newsvendor — Critical Ratio (Nahmias, 1982)

Q* = F⁻¹(C_u / (C_u + C_o))
C_u = stockout_factor × meal_price   (underage cost)
C_o = waste_multiplier × unit_cost   (overage cost)
F = Normal CDF of demand

Analytically optimal for single-period stochastic. Strong when cost ratio is calibrated. Fails when multi-day inventory dynamics (lead times, carry-over, FIFO aging) are important.

8.3 SMA + Reorder

Order when inventory ≤ threshold
Quantity = SMA(7d) × safety_factor − inventory

Common industry heuristic. Adapts to trend but ignores weekly cyclicality.

8.4 Seasonal Reorder (Added per faculty recommendation — strongest baseline)

ema[dow] = 0.3 × recent_demand + 0.7 × ema[dow]   # rolling per-DOW mean
dow_mult = {Fri: 1.6, Sat: 1.4, ...}               # restaurant-specific
order_trigger = inventory ≤ ema[dow] × dow_mult × lead_days
order_qty     = ema[dow] × dow_mult × safety_factor × (lead_days+1) − inventory

Uses the same temporal signal as RL agents but via a hand-crafted rule. Sets the highest practical bar for RL to beat. Achieves 43.9% FinPM and 61.7% SL in Scenario 1 — best service level of any method in the study.

9. Training Protocol — Step by Step

File: training_loop.py

9.1 Per-Episode Loop

for ep in range(n_episodes):
    state = env.reset()     # mode='train': variable length [20, ep_len_train]; random start [0, split_day)
    done = False
    while not done:
        action = agent.select_action(state, training=True)
        next_state, reward, done, info = env.step(action)
        agent.store(state, action, reward, next_state, done)
        agent.update()      # DQN/DDQN: gradient step each step
        state = next_state
    # Post-episode
    agent.decay_epsilon()   # Q-Learning, DQN, DDQN: ε × 0.995
    agent.update()          # PPO: compute returns, actor-critic update
    agent.end_episode()     # DQN/DDQN: sync target network every 20 eps

    if (ep + 1) % 100 == 0 and not is_ql:
        agent.decay_lr(ep+1, n_episodes, lr_min=0.0001)
        # lr = lr_min + 0.5×(lr₀−lr_min)×(1+cos(π×(ep+1)/n_episodes))

9.2 Training Hyperparameter Table

Parameter	Q-Learning	DQN	Double DQN	PPO
Episodes	800	600	600	500
LR₀	α=0.15 (fixed)	0.001	0.001	actor 0.001, critic 0.003
LR_min	—	0.0001	0.0001	actor 0.0001, critic 0.0003
LR decay	none	cosine/100 eps	cosine/100 eps	cosine/100 eps
γ	0.95	0.95	0.95	0.95
ε (start→end)	1.0→0.05	1.0→0.05	1.0→0.05	—
ε decay	0.995/ep	0.995/ep	0.995/ep	—
Hidden	—	128	128	128
Buffer	—	8,000	8,000	—
Batch	—	64	64	full episode
Target update	—	every 20 eps	every 20 eps	—
PPO clip ε	—	—	—	0.20
Entropy coef	—	—	—	0.01
Update epochs	—	1/step	1/step	6/episode
Eval episodes	100	100	100	100
Eval seeds	3	3	3	3

9.3 Variable Episode Length Rationale

Training with episode length sampled from Uniform[20, ep_len_train] prevents the agent from learning "in the last 5 days, I should stock up because the episode ends" — a horizon-specific artefact that would not generalise to deployment. Evaluation always uses a fixed 30-day horizon for fair comparison.

Per-restaurant ep_len_train: Kachchi Bhai=30, Star Kabab=45, Fakruddin=45, Madchef=40, Bismillah=30. Longer datasets get longer training episodes for better seasonal coverage.

9.4 Why Training Episodes Can Run Past split_day

Training start is in [0, split_day). For Kachchi Bhai with split_day=145:

Episode starting day 130 runs to day 160 — 9 days of Ramadan exposure (days 151–160)
Episode starting day 140 runs to day 170 — 19 days of Ramadan exposure

Without this, the maximum start would be 145 − 30 = 115 and no episode ever reaches Ramadan (day 151). The agent would be tested exclusively on Ramadan demand patterns it has never seen — causing the catastrophic −500% failures observed in earlier versions of this project.

10. Evaluation Protocol

10.1 Out-of-Sample Test Split

test_env = RestaurantInventoryEnv(cfg, empirical_csv=csv,
                                   seed=seed+1, mode='test')
# mode='test': start_day in [split_day, total_days − 30]
#              demand distributions from TRAINING data (statistically reliable)
#              episode length fixed at 30 days

All results come from agents running on calendar positions they were never explicitly trained on — the last 20% of each restaurant's historical period.

Why training distributions are used for evaluation too: Kachchi Bhai test split = 37 rows, which means ~5 data points per day-of-week for building test distributions. A distribution built from 5 points is unreliable. Using the training distributions (built from 145 data points, ~21 per day-of-week) gives stable demand parameters. The temporal generalisation is demonstrated by the episode start positions being in the test window, not by using a different demand model.

10.2 Multi-Seed Evaluation

100 evaluation episodes per agent
Episodes sampled across different start days within the test window
Metrics reported as: Mean ± Std across 100 episodes
Seeds: [42, 123, 777] used for agent re-initialisation in multi-seed mode

10.3 Metrics

Metric	Formula	What It Reveals
Total Reward	Σ raw_reward over episode	Agent's optimisation objective
Financial PM	(Rev − OrderCost − WasteCost − HoldingCost) / Rev × 100	Cash profit — what accountant reports
RL-Adjusted PM	(Rev − all costs incl. stockout) / Rev × 100	Reward alignment — penalises lost customers
Service Level ± Std	Served / Demanded × 100	Customer satisfaction; operational performance
Waste %	Wasted / (InitialInv + Received) × 100	Spoilage efficiency (corrected denominator)
Stockout Rate	100 − SL	Demand not met

11. Experimental Scenarios

11.1 Scenario 1 — Generic Restaurant (USD, no cultural events)

Purpose: clean controlled benchmark, pure algorithmic comparison with no BDT-scale or Ramadan complications.

Ingredients: Rice ($0.50, 7d), Chicken ($3.00, 2d), Onion ($0.20, 5d),
             Cooking Oil ($1.00, 30d), Spice Mix ($0.80, 60d)
Initial inv: [60, 40, 50, 60, 50]
Meals:       Rice Bowl ($8), Grilled Chicken ($12), Stir-Fry ($10),
             Curry ($11), Veg Soup ($6)
Base demand: [20, 18, 15, 22, 12] orders/day | Demand std: 25%
Meal network:
  Rice Bowl:       3 Rice + 0.5 Spice Mix
  Grilled Chicken: 2 Chicken + 0.5 Spice Mix + 0.3 Cooking Oil
  Stir-Fry:        1 Chicken + 1 Onion + 0.5 Cooking Oil + 0.3 Spice Mix
  Curry:           2 Chicken + 1 Onion + 1 Spice Mix + 0.5 Cooking Oil
  Veg Soup:        1 Onion + 0.5 Spice Mix
Virtual calendar: 600 days | Train: 0–480 | Test: 480–600
Weekend boost: 1.35× | Seasonal amplitude: 10% | Lead time: 1d | Reward scale: 1.0

11.2 Scenario 2 — Five Bangladeshi Restaurants (BDT)

Key environmental distinctions from Scenario 1:

BDT currency; meal prices BDT 70–550; ingredient costs BDT 15–950/unit
Ramadan/Eid calendar overlays
Demand from real empirical distributions (historical CSV), not parametric
Hilsha Fish at Fakruddin: 1-day shelf life — most extreme perishability in study
Madchef: 35% demand std — most volatile environment
Bismillah: BDT 70–160 meal prices create structural low-margin challenge
2-day lead time at Fakruddin (Gulshan supply chain)

12. Scenario 1 Results — Generic Restaurant

Test split: virtual days 480–600. 100 evaluation episodes.

Agent	Type	Reward	Fin PM	RL-Adj PM	Service Level	Waste %
DQN ★	RL	−13,746	51.5%	−207.2%	29.6 ± 1.6%	7.0%
Double DQN	RL	−11,488	30.9%	−116.3%	41.0 ± 2.2%	4.1%
Q-Learning †	RL	−11,714	26.3%	−119.5%	40.8 ± 3.7%	5.7%
PPO	RL	−12,751	19.3%	−127.4%	36.8 ± 1.1%	0.9%
Seasonal Reorder	BL	−2,139	43.9%	−13.1%	61.7 ± 1.8%	2.5%
SMA + Reorder	BL	−12,268	27.4%	−130.7%	35.4 ± 2.0%	11.7%
Newsvendor	BL	−16,273	6.8%	−234.6%	29.4 ± 1.4%	20.9%
EOQ	BL	−16,888	6.3%	−261.3%	27.1 ± 1.5%	21.7%

★ Best RL on Financial PM | † Low-capacity benchmark

Headline: DQN (51.5%) beats all classical baselines including Seasonal Reorder (43.9%) on Financial PM — RL wins outright on the primary cash profitability metric, evaluated on held-out future data.

Agent analysis:

DQN (51.5%): Learned aggressive ordering on days before weekends + conservative ordering on Monday-Tuesday when Chicken has 2d SL and lower demand. The 7.0% waste reflects occasional over-ordering; the 29.6% SL shows some unmet demand; net financial margin is the best.
Double DQN (30.9%, 41.0% SL): Lower margin than DQN but serves more customers. Decoupled update produces a more conservative policy — less overestimation-driven over-ordering, but also less aggressive revenue capture.
Q-Learning (26.3%, 40.8% SL): Competitive in this simple USD environment because the weekly cycle is capturable in 7,168 bins. The information loss from discarding age ratios is not critical here.
PPO (19.3%, 0.9% waste): Lowest waste of any agent — entropy regularisation and Monte Carlo returns produce a risk-averse policy that essentially never over-orders. Serves only 36.8% of demand.
Seasonal Reorder (43.9%, 61.7% SL): Best SL by explicit day-of-week rule. Its RL-Adjusted PM of −13.1% is the best of any method — it balances stockout and waste with high accuracy.

13. Scenario 2 Results — Five Bangladeshi Restaurants

All evaluated on held-out test splits. 100 evaluation episodes.

13.1 Kachchi Bhai — Test = Ramadan/Eid (days 145–182)

Agent	Reward	Fin PM	RL-Adj PM	SL	Waste %
Double DQN ★	−1,607,004	25.5%	−500.0%	0.4%	32.7%
DQN	−1,614,113	24.7%	−500.0%	0.4%	33.1%
Q-Learning †	−2,599,173	−462.6%	−500.0%	11.5%	7.7%
PPO	−2,674,716	−500.0%	−500.0%	11.3%	13.2%
Seasonal Reorder	−3,459,013	−417.3%	−500.0%	24.0%	0.5%
Newsvendor	−3,194,726	−481.4%	−500.0%	16.1%	8.9%

RL wins: Double DQN +25.5% vs best BL −417.3% (442 pp margin)

13.2 Star Kabab — Best Single Result in Scenario 2

Agent	Reward	Fin PM	RL-Adj PM	SL	Waste %
Double DQN ★	−1,353,919	72.6%	−500.0%	0.4%	28.5%
DQN	−1,495,727	−500.0%	−500.0%	0.4%	74.0%
Q-Learning †	−2,269,254	−443.0%	−500.0%	9.8%	28.0%
PPO	−2,464,993	−500.0%	−500.0%	5.5%	45.9%
Seasonal Reorder	−3,041,490	−377.0%	−500.0%	21.6%	20.6%
Newsvendor	−2,628,810	−439.0%	−500.0%	13.8%	25.7%

RL wins: Double DQN +72.6% vs best BL −377.0% (449 pp margin)

DQN 74% waste vs Double DQN 28.5% — the starkest demonstration of overestimation collapse in the study.

13.3 Fakruddin — Hilsha Fish Environment

Agent	Reward	Fin PM	RL-Adj PM	SL	Waste %
Double DQN ★	−1,425,837	−142.9%	−500.0%	0.7%	26.3%
DQN	−1,428,236	−143.4%	−500.0%	0.7%	26.6%
PPO	−3,550,755	−500.0%	−500.0%	33.7%	14.6%
Q-Learning †	−3,467,016	−500.0%	−500.0%	21.0%	17.1%
Seasonal Reorder	−5,706,467	−500.0%	−500.0%	41.5%	15.9%
Newsvendor	−4,178,832	−500.0%	−500.0%	20.3%	20.8%

RL wins: Double DQN −142.9% vs best BL −500.0% (357 pp margin)

All baselines hit −500% floor. Hilsha Fish (1d SL, 2d lead time) makes this environment the hardest in the study.

13.4 Madchef — Classical Analysis Wins

Agent	Reward	Fin PM	RL-Adj PM	SL	Waste %
DQN	−1,705,764	−500.0%	−500.0%	0.3%	89.4%
Double DQN	−1,767,659	−500.0%	−500.0%	0.3%	92.0%
PPO ★	−1,987,715	−231.4%	−500.0%	8.3%	38.4%
Q-Learning †	−2,005,386	−255.1%	−500.0%	12.0%	29.5%
Newsvendor ★★	−2,079,838	−200.0%	−500.0%	18.7%	20.6%

Newsvendor wins: −200.0% vs best RL PPO −231.4%

DQN/DDQN overestimation collapse — 89–92% waste when all ingredients are 2-day perishables.

13.5 Bismillah Hotel — Low-Margin Challenge

Agent	Reward	Fin PM	RL-Adj PM	SL	Waste %
DQN ★	−562,827	−29.1%	−500.0%	0.4%	13.5%
Double DQN	−561,044	−35.4%	−500.0%	0.4%	13.8%
Q-Learning †	−1,230,382	−500.0%	−500.0%	7.1%	6.4%
PPO	−1,789,271	−500.0%	−500.0%	0.4%	62.8%
Newsvendor	−1,039,880	−448.2%	−500.0%	11.2%	0.7%

RL wins: DQN −29.1% vs best BL −448.2% (419 pp margin)

13.6 Cross-Restaurant Summary

Restaurant	Best RL	Best RL FinPM	Best BL FinPM	RL vs BL
Kachchi Bhai	Double DQN	+25.5%	−417.3% (Seasonal Reorder)	RL wins +442 pp
Star Kabab	Double DQN	+72.6%	−377.0% (Seasonal Reorder)	RL wins +449 pp
Fakruddin	Double DQN	−142.9%	−500.0% (all)	RL wins +357 pp
Madchef	PPO	−231.4%	−200.0% (Newsvendor)	BL wins +31 pp
Bismillah	DQN	−29.1%	−448.2% (Newsvendor)	RL wins +419 pp

RL outperforms best classical baseline at 4 out of 5 restaurants.

14. Cross-Restaurant Transfer Test

Agents trained on Star Kabab + Fakruddin + Madchef + Bismillah evaluated zero-shot on Kachchi Bhai test split:

Agent	Fin PM	RL-Adj PM	SL	Waste %
Double DQN ★	27.2%	−500.0%	0.4%	33.1%
DQN	22.0%	−500.0%	0.4%	33.4%
Q-Learning	−479.7%	−500.0%	11.4%	8.1%
PPO	−500.0%	−500.0%	7.0%	9.8%

Double DQN: 27.2% FinPM zero-shot on unseen restaurant — nearly identical to the in-distribution result (25.5%). The conservative ordering policy learned across four BDT environments with perishable ingredients transfers directly to a fifth unseen restaurant with different cuisine, menu, prices, and schema.

15. Key Findings, Reasons & Insights

Finding 1 — DQN Wins Scenario 1 Financial PM on Future Data

Result: DQN 51.5% FinPM on test split (virtual days 480–600).
Why: DQN's replay buffer breaks temporal correlations; target network stabilises Q-targets. The agent learned the weekend demand cycle from training data (days 0–480) and applies it to evaluation (days 480–600) — a different slice of the virtual calendar.
Significance: Proves generalisation to future demand, not memorisation of training sequences.

Finding 2 — Double DQN Wins 3/5 Scenario 2 Restaurants

Result: DDQN average FinPM: −116.0%. DQN average: −229.6%.
Why: Kachchi Bhai, Star Kabab, and Fakruddin all have highly asymmetric over/under-ordering costs for perishable proteins. DQN's overestimation bias systematically drives excessive protein ordering; DDQN's decoupled update corrects this.
Significance: Empirically confirms the theoretical advantage of DDQN on cost-asymmetric perishable inventory.

Finding 3 — Q-Learning Degrades on Scenario 2

Result: Q-Learning: 40.8% SL in Scenario 1 (competitive); fails across all Scenario 2 restaurants.
Why: Scenario 2 requires capturing FIFO age ratios, demand EMA shifts (Ramadan signal), and ingredient-level inventory simultaneously. These interactions require the full 17-dim continuous state. Q-Learning's 7,168-bin discretisation discards age ratios and EMA entirely.
Significance: Demonstrates empirically why continuous function approximation is necessary for realistic perishable inventory.

Finding 4 — RL Beats All Baselines in Scenario 1 Financial PM

Result: DQN 51.5% > Seasonal Reorder 43.9% by 7.6 percentage points.
Why: Seasonal Reorder applies a fixed DOW multiplier. DQN learns richer interactions — given specific inventory level, batch age, and demand EMA, what is the exact profitable quantity? The continuous state enables exploiting interactions no hand-crafted rule can express.
Significance: Primary quantitative finding — a trained RL agent outperforms the strongest practical heuristic on out-of-sample financial performance.

Finding 5 — Train/Test Split Validates Generalisation

Result: All results evaluated on held-out future data (last 20% of each restaurant's history).
Why: This is a harder standard than evaluating on the same distribution trained on. It directly validates the deployment claim — the agent works on data it has never seen.
Significance: Higher standard than most RL inventory literature, which evaluates on training-distribution data.

Finding 6 — Zero-Shot Transfer: Double DQN 27.2% FinPM

Result: Agents trained on 4 restaurants achieve 27.2% FinPM on unseen Kachchi Bhai.
Why: The conservative ordering strategy (order minimally to avoid waste; serve initial stock at premium prices) is structurally optimal across BDT perishable environments — it exploits a property of the MDP, not a restaurant-specific pattern.
Significance: Architecture-agnostic generalisation across cuisines, schemas, and demand regimes without fine-tuning.

Finding 7 — Dual PM Metrics Reveal Hidden Policy Failures

Result: Star Kabab Double DQN shows +72.6% Financial PM and −500% RL-Adjusted PM simultaneously.
Why: The agent serves 0.4% of demand (near-zero SL) generating small positive cash margins. RL-Adjusted PM includes the stockout penalty on the 99.6% unserved demand, revealing the true cost.
Significance: Proves the necessity of dual metrics — either metric alone would completely mischaracterise this policy.

Finding 8 — Madchef: Classical Analysis Beats Learned RL

Result: Newsvendor −200% outperforms all RL agents at Madchef.
Why: Madchef has 35% demand variance, the highest in the study. The analytical Newsvendor critical ratio is calibrated to this variance; DQN/DDQN's overestimation cascade produces 89–92% waste when all primary ingredients share 2-day shelf lives.
Significance: Honest finding about RL limits — classical analysis wins when variance is high and inventory economics are extreme.

Finding 9 — Conservative Ordering Finds Positive Financial Margin

Result: DQN/DDQN achieve positive FinPM at Kachchi Bhai (24–25%), Star Kabab (72.6%), Bismillah (−29.1%) with SL of 0.4%.
Why: Serving very few customers at BDT 320–520/serving with near-zero ordering cost and near-zero waste generates slight positive cash margin. The RL-Adjusted PM (−500%) confirms this is operationally unsustainable.
Significance: Shows why the reward signal (RL-Adjusted PM) and business KPI (Financial PM) must be clearly distinguished and both reported.

Finding 10 — Framework Generalises Across Five Culturally Distinct Restaurants

Result: Same code, agents, and training loop produced meaningful results at all 5 restaurants.
Why: The empirical demand loader adapts to each CSV schema automatically. The config-driven MDP parameterises all restaurant-specific properties without code changes. The bipartite network formulation is universal.
Significance: Architecture-agnostic cross-cultural robustness — from Biryani/Mughlai to Modern Fast Food, from 182-row CSV to 120,607-row item log.

16. Research Contributions

Hybrid RL-SNO Framework: First unified framework jointly modelling the ingredient-meal bipartite supply network and learning ordering policies through deep RL — treating network coupling as a first-class constraint.
Industry-Realistic MDP: FIFO batch aging + 1–2 day lead times + ingredient-specific action levels + calendar-offset episodes + variable training length + reward normalisation = most operationally complete restaurant inventory MDP in the RL literature.
Four Agents from First Principles: Pure-NumPy Q-Learning, DQN, Double DQN, PPO demonstrating backpropagation, experience replay, and score-function policy gradients at implementation level — no deep learning frameworks.
Corrected Waste Metric: waste / (initial_inv + received) rather than waste / max(ordered, 1) — prevents inflation when an agent orders near-zero. A methodological contribution applicable to any perishable inventory RL study.
Dual Profit-Margin Metrics: Financial PM + RL-Adjusted PM cleanly separates cash performance from reward alignment. Enables precise detection of degenerate policies that appear profitable while systematically failing customers.
Train/Test Temporal Split: 80/20 chronological split on all datasets; evaluation exclusively on held-out future data — a stricter standard than most RL inventory papers.
Zero-Shot Cross-Restaurant Transfer: Train on 4 restaurants, evaluate on 5th unseen. Double DQN 27.2% FinPM zero-shot — architecture-agnostic generalisation without fine-tuning.
Five-Restaurant Primary Dataset: Operational data from 5 culturally distinct Bangladeshi restaurants across 4 cuisines, 5 Dhaka districts, schemas from 182 rows to 120,607 rows, 3,467 combined operational days.
29-Test Reproducible Evaluation Suite: FIFO spoilage, waste denominator, train/test split, variable episode, reward normalisation, LR decay, KPI computation, action decoding, empirical demand, calendar offset — all verified.

17. Limitations & Future Work

Limitations

Kachchi Bhai test window = Ramadan/Eid: 182-day dataset with Ramadan at day 151; 80/20 split puts all Ramadan in test window. A 2-year dataset would allow Ramadan in both splits.
Slow convergence on BDT: 600 episodes insufficient for some environments. 2,000+ episodes with larger replay buffers would produce more stable policies.
DQN overestimation at Madchef: 89–92% waste when all ingredients share 2d SL. The cascade cannot be fully prevented at 600 episodes without algorithmic modifications.
No Ramadan feature in state: agents infer Ramadan from demand EMA (takes ~3 days to shift). Adding days_into_ramadan / 30 to the 17-dim state would dramatically improve Ramadan-period performance.
Fixed ingredient costs: BDT Mutton and Hilsha prices fluctuate significantly around Eid and Bengali New Year.

Future Work

Add Ramadan/Eid calendar features to the 17-dim state vector
Multi-restaurant joint policy: single agent trained on all 5 simultaneously
PPO with Generalised Advantage Estimation (GAE) for lower-variance policy gradients
Extend to 2-year datasets so Ramadan appears in both train and test splits
Online continual learning: fine-tune on incoming daily data
REST API deployment wrapper for POS system integration

18. How to Run

pip install -r requirements.txt

python main.py              # full pipeline (~15 min, all 5 restaurants)
python main.py --quick      # demo (~3.5 min, Kachchi Bhai only for S2)
python main.py --s1only     # Scenario 1 only
python main.py --s2only     # Scenario 2 only
python main.py --tests      # run 29 unit tests
python main.py --figs       # regenerate all figures from saved results

Outputs

results/
├── eda/                       (16 EDA figures)
├── scenario1/                 (4 RL figures + CSVs + JSON metrics)
├── scenario2/
│   ├── {restaurant}/          (8 figures + CSVs per restaurant)
│   ├── cross_restaurant_transfer.csv
│   └── scenario2_combined.csv
├── CS_1_cross_scenario_kpi.png
├── CS_2_rl_improvement.png
├── CS_3_cross_restaurant_pm.png
└── COMBINED_RESULTS.csv

Programmatic

from config import KACCHI_BHAI, TRAIN_CONFIG, STATE_DIM, build_action_space
from restaurant_env import RestaurantInventoryEnv
from rl_agents import build_agents
from training_loop import train_agent, evaluate_agent

_, order_matrix = build_action_space(KACCHI_BHAI)
csv = 'data/kacchi_bhai_daily_orders.csv'
train_env = RestaurantInventoryEnv(KACCHI_BHAI, empirical_csv=csv, seed=42, mode='train')
test_env  = RestaurantInventoryEnv(KACCHI_BHAI, empirical_csv=csv, seed=43, mode='test')

agents = build_agents(STATE_DIM, 7168, train_env.n_actions, TRAIN_CONFIG)
train_agent(train_env, agents['DQN'], n_episodes=600, agent_name='DQN')
mean_m, _, _ = evaluate_agent(test_env, agents['DQN'], n_episodes=100)
print(f"FinPM: {mean_m['financial_pm']:.1f}% | SL: {mean_m['service_level']:.1f}%")

20. Requirements

numpy>=1.24
pandas>=1.5
matplotlib>=3.6
scipy>=1.10
seaborn>=0.12
openpyxl>=3.1
pytest>=7.0

Python 3.9+. No deep learning frameworks required.

Name		Name	Last commit message	Last commit date
data		data
results		results
tests		tests
.gitattributes		.gitattributes
README.md		README.md
classical_baselines.py		classical_baselines.py
config.py		config.py
data_loader.py		data_loader.py
eda_scenario1.py		eda_scenario1.py
eda_scenario2.py		eda_scenario2.py
main.py		main.py
requirements.txt		requirements.txt
restaurant_env.py		restaurant_env.py
results_summary.py		results_summary.py
rl_agents.py		rl_agents.py
scenario1_generic.py		scenario1_generic.py
scenario2_kacchi_bhai.py		scenario2_kacchi_bhai.py
training_loop.py		training_loop.py
visualisations.py		visualisations.py

Latest commit

History

Folders and files

Repository files navigation