Skip to content

Commit c9dd689

Browse files
Pranavk098Pranavk098
committed
Changes in README - mermaid for architecture added few sections. added CI
1 parent 4cf62c2 commit c9dd689

4 files changed

Lines changed: 98 additions & 33 deletions

File tree

‎.github/workflows/ci.yml‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
name: CI
2+
3+
on:
4+
push:
5+
branches: ["main"]
6+
pull_request:
7+
branches: ["main"]
8+
9+
jobs:
10+
lint-and-dry-run:
11+
runs-on: ubuntu-latest
12+
13+
steps:
14+
- uses: actions/checkout@v4
15+
16+
- uses: actions/setup-python@v5
17+
with:
18+
python-version: "3.10"
19+
cache: "pip"
20+
21+
- name: Install core dependencies
22+
run: pip install -e ".[eval]"
23+
24+
- name: Check imports
25+
run: python -c "from src.pipeline.orchestrator import main; from src.api.server import app; from src.continual_learning.ewc import EWC"
26+
27+
- name: Dry-run pipeline
28+
run: python -m src.pipeline.orchestrator --stage all --image product.jpg --dry-run

‎COVER_LETTER.md‎

Lines changed: 5 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -3,9 +3,6 @@
33

44
---
55

6-
> *"The best engineers don't just solve the problem in front of them —
7-
> they solve the problem behind the problem."*
8-
96
This repository is my application to join Zippin.
107

118
Not a resume. Not a cover letter template. A working proof — built on my own time — that I understand your core engineering bottleneck deeply enough to have already started solving it.
@@ -32,6 +29,8 @@ When you fine-tune a detection model on new SKU data, it overwrites the paramete
3229
My thesis: **these three constraints are not independent problems — they are one problem**.
3330
And the solution is a single, disciplined pipeline that addresses all three simultaneously.
3431

32+
> **The detail that separates this from a paper re-implementation:** Most public EWC code computes the Fisher matrix and stops — it measures importance without ever injecting the penalty into training. This codebase overrides `criterion()` in a custom `DetectionTrainer` subclass so the EWC penalty lands on every gradient update, not just around it. The math is real. The forgetting protection is real. Full breakdown in Decision 2 below.
33+
3534
---
3635

3736
## The Architecture Decisions — And Why I Made Them
@@ -155,19 +154,15 @@ The most important thing in a complex codebase is understanding *why* a choice w
155154

156155
Before reaching out, I spent real time studying Zippin from the outside: engineering blog posts, patent filings, published architecture descriptions, conference talks from your Chief Scientist. I read everything publicly available about your edge compute constraints, your sensor fusion approach, and the specific challenges of deploying frictionless checkout across heterogeneous venue formats.
157156

158-
Then I spent approximately 40 hours writing code against those constraints — not generic ML code, but code that respects the 8GB VRAM ceiling, the 60 FPS SLA, and the continual learning problem that gets harder with every SKU you add to the catalogue. The three future proposals in this repository (`sensor_fusion.py`, `analytics_rag.py`, the EWC benchmark) are not whiteboard ideas. They are running implementations of architectural solutions to problems I identified in your published technical writing.
157+
Then I built code against those constraints — not generic ML code, but code that respects the 8GB VRAM ceiling, the 60 FPS SLA, and the continual learning problem that gets harder with every SKU you add to the catalogue. The three future proposals in this repository (`sensor_fusion.py`, `analytics_rag.py`, the EWC benchmark) are not whiteboard ideas. They are running implementations of architectural solutions to problems I identified in your published technical writing.
159158

160159
That is what I'm willing to invest before I'm even on the team.
161160

162-
**What I'm asking for is a chance — one conversation.**
161+
**I'm asking for one conversation.**
163162

164163
A 30-minute call with Motilal Agrawal, Abhinav Katiyar, or whoever on the engineering team is the right person to evaluate this work. I want to walk through the architecture decisions, answer the hard questions (why EWC over QLoRA, what breaks at 10,000 SKUs, how the EKF degrades under multi-shopper occlusion, where the Sim2Real gap actually lives), and let the quality of the technical judgment speak for itself.
165164

166-
I am not asking for a favour. I am asking for the opportunity to prove — in a real engineering conversation — that the depth of investment you're seeing here carries forward into every problem I'd touch on the team.
167-
168-
I am genuinely excited about what Zippin is building. Autonomous retail at stadium scale, with the accuracy requirements and edge constraints you're operating under, is one of the most technically interesting deployment problems in applied AI right now. I want to be in the room where those problems get solved.
169-
170-
If this work demonstrates the kind of engineering judgment that belongs on your team, I'd very much welcome the conversation.
165+
Autonomous retail at stadium scale, with the accuracy requirements and edge constraints you're operating under, is one of the most technically interesting deployment problems in applied AI right now. I want to be in the room where those problems get solved.
171166

172167
---
173168

‎LICENSE‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
MIT License
2+
3+
Copyright (c) 2025 Pranav Koduru
4+
5+
Permission is hereby granted, free of charge, to any person obtaining a copy
6+
of this software and associated documentation files (the "Software"), to deal
7+
in the Software without restriction, including without limitation the rights
8+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9+
copies of the Software, and to permit persons to whom the Software is
10+
furnished to do so, subject to the following conditions:
11+
12+
The above copyright notice and this permission notice shall be included in all
13+
copies or substantial portions of the Software.
14+
15+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21+
SOFTWARE.

‎README.md‎

Lines changed: 44 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,7 @@
11
# Zero-Shot SKU Onboarding — Zippin Edge AI Platform (v2.0)
22

3+
![Python](https://img.shields.io/badge/python-3.10%2B-blue) ![License](https://img.shields.io/badge/license-MIT-green) ![CI](https://github.com/pranavkoduru/zippin-synthetic-onboarding-poc/actions/workflows/ci.yml/badge.svg)
4+
35
A production-grade, end-to-end pipeline that onboards a **new retail SKU in under 10 minutes** from a single product photograph — no real-world data collection required.
46

57
---
@@ -19,29 +21,16 @@ Manual dataset collection for every new shelf item makes rapid Zippin store expa
1921

2022
## Architecture
2123

22-
```
23-
┌─────────────────────────────────────────────────────────────────────┐
24-
│ POST /onboard (FastAPI) │
25-
│ ┌────────────────────────┐ │
26-
│ product.jpg ───► │ Stage 1: VLM Extract │ Ollama + LLaVA-7B │
27-
│ │ (semantic attributes) │ 4-bit GGUF, <8GB VRAM│
28-
│ └──────────┬─────────────┘ │
29-
│ │ shape, material, colors │
30-
│ ┌──────────▼─────────────┐ │
31-
│ │ Stage 2: BlenderProc2 │ 50 renders/SKU │
32-
│ │ (COCO synthetic data) │ HDRI + occlusion DR │
33-
│ └──────────┬─────────────┘ │
34-
│ │ COCO JSON │
35-
│ ┌──────────▼─────────────┐ │
36-
│ │ Stage 3: YOLOv8n │ ~3.2M params │
37-
│ │ + EWC (Fisher matrix) │ 60 FPS Jetson Orin │
38-
│ └──────────┬─────────────┘ │
39-
│ │ fine-tuned .pt │
40-
│ ┌──────────▼─────────────┐ │
41-
│ │ Stage 4: Sim2Real Eval│ pycocotools mAP@50 │
42-
│ │ (real shelf images) │ confidence breakdown │
43-
│ └────────────────────────┘ │
44-
└─────────────────────────────────────────────────────────────────────┘
24+
```mermaid
25+
flowchart TD
26+
I[product.jpg] --> A
27+
subgraph API ["POST /onboard (FastAPI async job queue)"]
28+
A["Stage 1: VLM Extract\nOllama + LLaVA-7B GGUF\n~4.2 GB VRAM"] -->|"shape · material · colors"| B
29+
B["Stage 2: BlenderProc2\n50 renders · HDRI + occlusion DR\nCOCO annotations"] -->|"coco_annotations.json"| C
30+
C["Stage 3: YOLOv8n + EWC\nFisher matrix per-batch\n~25 MB state overhead"] -->|"fine-tuned .pt"| D
31+
D["Stage 4: Sim2Real Eval\npycocotools mAP@50\nconfidence breakdown"]
32+
end
33+
D --> E["GET /skus/{id}/metrics"]
4534
```
4635

4736
---
@@ -90,6 +79,38 @@ Physics-based rendering with:
9079

9180
---
9281

82+
## Demo Output
83+
84+
### Synthetic renders (BlenderProc2 — domain-randomised)
85+
86+
Generated from a single product photo. 50 renders/SKU with randomised HDRI lighting, occluders, and camera angles:
87+
88+
| | | |
89+
|---|---|---|
90+
| ![render 0](checkpoints/synthetic_dataset/images/000000.jpg) | ![render 1](checkpoints/synthetic_dataset/images/000005.jpg) | ![render 2](checkpoints/synthetic_dataset/images/000010.jpg) |
91+
| ![render 3](checkpoints/synthetic_dataset/images/000015.jpg) | ![render 4](checkpoints/synthetic_dataset/images/000020.jpg) | ![render 5](checkpoints/synthetic_dataset/images/000025.jpg) |
92+
93+
### Training run results (YOLOv8n + EWC)
94+
95+
![Training results](runs/detect/checkpoints/yolo_run/results.png)
96+
97+
| Validation labels | Validation predictions |
98+
|---|---|
99+
| ![val labels](runs/detect/checkpoints/yolo_run/val_batch0_labels.jpg) | ![val preds](runs/detect/checkpoints/yolo_run/val_batch0_pred.jpg) |
100+
101+
### Dry-run (no GPU required)
102+
103+
```bash
104+
python -m src.pipeline.orchestrator --stage all --image product.jpg --dry-run
105+
# [DRY-RUN] Stage 1: VLM extract skipped
106+
# [DRY-RUN] Stage 2: BlenderProc2 skipped (would generate 50 renders)
107+
# [DRY-RUN] Stage 3: EWC fine-tune skipped
108+
# [DRY-RUN] Stage 4: Eval skipped
109+
# Pipeline wiring: OK
110+
```
111+
112+
---
113+
93114
## Repository Structure
94115

95116
```

0 commit comments

Comments
 (0)