VisionCart is a visual checkout prototype that recognizes grocery products from a live camera or uploaded image, builds a reviewable basket, calculates prices with Decimal-safe billing, and produces a receipt. The application combines a fine-tuned YOLOv8s detector with a FastAPI backend and a custom React + TypeScript retail interface.
On 494 untouched test scenes across 25 grocery classes, the selected YOLO-only path reached 0.8363 item F1, 57.09% exact-basket accuracy, and $1.6872 billing MAE.
A curated multi-product checkout scene demonstrates the production VisionCart flow. The frozen YOLO-only path detects oil and tea and produces an $8.48 basket; the visible honey jar remains undetected. This example is for qualitative demonstration only and is not used in reported evaluation metrics. See the qualitative input, annotated result, and provenance record.
VisionCart detects three milk products in a held-out Freiburg Groceries test scene. The three detections and $5.97 basket were produced by the same frozen application path. This image belongs to the final test domain, but the reported aggregate metrics come from the complete test split—not this single scene. See the validated input, annotated result, and selection record.
- Detects and counts grocery products across 25 trained classes with YOLOv8s.
- Supports live browser camera capture, drag-and-drop, and image-upload checkout.
- Surfaces uncertain detections for review instead of silently forcing a product label.
- Recalculates edited quantities and prices through the backend's Decimal billing engine.
- Produces a receipt and structured transaction log after checkout.
- Offers an optional browser-based spoken receipt with replay and stop controls.
- Uses a held-out comparison—not architectural complexity—to choose YOLO-only over YOLO + ResNet-50.
flowchart LR
A[Camera or image<br/>React + TypeScript] --> B[FastAPI adapter]
B --> C[YOLOv8s detection]
C --> D[Confidence and<br/>catalog validation]
D --> E[Decimal pricing<br/>and billing]
E --> F[Basket, receipt,<br/>transaction log]
F --> G[Optional browser speech]
YOLOv8s performs localization, classification, and quantity-producing detection in the final application path. FastAPI keeps inference, validation, pricing, billing, and transaction logging authoritative on the backend while the React client handles capture, overlays, review, and receipt presentation.
A ResNet-50 crop classifier was evaluated as an experimental second stage. It reduced item F1 from 0.8363 to 0.7160 and increased median CPU latency from 150.48 ms to 267.43 ms, so the simpler YOLO-only path became the production default. The classifier experiment remains reproducible evidence, not part of the default checkout architecture.
All results below use the untouched Freiburg Groceries v10 test split. Detection thresholds were selected on validation data and frozen before final evaluation.
| Metric | Result |
|---|---|
| Precision | 0.8369 |
| Recall | 0.8379 |
| mAP@50 | 0.9038 |
| mAP@50:95 | 0.7631 |
| Metric | Result |
|---|---|
| Item F1 | 0.8363 |
| Quantity accuracy | 60.53% |
| Exact basket accuracy | 57.09% |
| Billing MAE | $1.6872 |
Exact-basket accuracy is intentionally reported alongside item F1: a checkout scene is exact only when every product and quantity is correct. This stricter result makes the prototype's current readiness and remaining error rate visible.
| Metric | YOLO-only (selected) | YOLO + ResNet-50 |
|---|---|---|
| Item F1 | 0.8363 | 0.7160 |
| Exact basket | 57.09% | 42.71% |
| Billing MAE | $1.6872 | $2.2940 |
| Median CPU latency | 150.48 ms | 267.43 ms |
YOLO-only was retained because it was both more accurate and substantially faster on the same 494 held-out scenes. Its measured throughput was 6.65 FPS versus 3.74 FPS for the two-stage path on the documented CPU benchmark. For context, the historical ResNet-50 classifier achieved 76.31% top-1 accuracy and 0.7391 macro F1 on 1,199 held-out ground-truth crops, but those crop-level results did not translate into a stronger checkout pipeline.
The responsive interface separates the shopper flow from optional developer detail. A user can capture a camera frame or upload an image, inspect labeled detection overlays, adjust product quantities, resolve low-confidence items, and complete checkout only after review. Totals are recalculated by the backend rather than trusted from browser state.
The completed view presents line items, quantities, unit prices, total, timestamp, and transaction reference. Receipt speech is optional and browser-based; checkout remains functional if speech is unsupported. Developer view exposes the active model, architecture, threshold, and inference latency without adding those details to the default shopper experience.
VisionCart uses Freiburg Groceries v10 on Roboflow, with 25 grocery categories.
| Split | Images | Boxes | Classifier crops |
|---|---|---|---|
| Train | 3,946 | 9,231 | 9,231 |
| Validation | 493 | 1,178 | 1,178 |
| Test | 494 | 1,199 | 1,199 |
| Total | 4,933 | 11,608 | 11,608 |
Classifier crops were generated from ground-truth boxes and class IDs, with source split and SHA-256 provenance retained in the crop manifest. Dataset validation found zero SHA-256 cross-split duplicate images, all 25 classes in every split, and no skipped crops. Confidence thresholds were selected using validation scenes only; the final detector, classifier, end-to-end, and architecture-comparison metrics were measured on untouched test data.
The full methodology and environment record are in the validation summary.
| Area | Technologies |
|---|---|
| ML and vision | Python, PyTorch, Ultralytics YOLOv8, OpenCV, torchvision |
| Backend | FastAPI, Pydantic, Uvicorn, Decimal billing |
| Frontend | React, TypeScript, Vite, browser Media and Speech APIs |
| Data and integrations | Local CSV pricing, JSONL transactions, optional Google Sheets adapters |
| Validation | pytest, Vitest, Ruff, ESLint, TypeScript, compileall |
Current repository validation passes 62 Python tests and 14 frontend tests, along with Ruff, Python compilation, TypeScript checking, ESLint, and the Vite production build.
Python 3.10 or newer is required. Install the application and web dependencies, then build the frontend:
python -m pip install -e ".[app,web]"
cd frontend
npm install
npm run build
cd ..Launch the complete checkout application:
python app.py --uiOpen http://127.0.0.1:8000. For single-image CLI inference:
python app.py --image path/to/checkout.jpgInference expects the validated detector checkpoint at models/yolo_best.pt. Model binaries and raw/processed datasets are intentionally excluded by .gitignore; the dataset downloader uses ROBOFLOW_API_KEY from the environment or a private root .env. The included .env.example contains safe placeholders, and the local price catalog works without Google credentials.
With the dataset and validated checkpoints present, reproduce validation, threshold checks, held-out evaluation, architecture comparison, benchmarks, failure artifacts, and reports with:
python scripts/run_full_pipeline.py --skip-training --device autoInstall the additional evaluation dependencies with python -m pip install -e ".[ml,dev]" before running the pipeline. Remove --skip-training only when a full training run is intended; focused training and evaluation entry points are available under scripts/. See the validation summary for the exact measured environment and results.
visioncart/
├── frontend/ React and TypeScript checkout interface
├── src/visioncart/ Inference, billing, integrations, and FastAPI adapter
├── scripts/ Data, training, evaluation, and reporting workflows
├── tests/ Python application and regression tests
├── reports/ Machine-readable metrics and validation evidence
├── assets/ UI captures, plots, and held-out failure cases
├── configs/ Production defaults and thresholds
└── app.py Repository-root CLI and UI entry point
- Exact-basket accuracy is 57.09%; a single missed, duplicated, or misclassified item makes the scene incorrect.
- Recognition is closed-set to the 25 Freiburg classes and does not safely identify unknown products.
- Domain shift to current retail packaging, store lighting, checkout hardware, and broader inventory has not been measured.
- Checkout uses a captured scene rather than persistent multi-frame object tracking.
- The latency benchmark is CPU- and workload-specific; payment and inventory-system integrations are outside this prototype.
- Validation summary — dataset integrity, methodology, metrics, and measured environment
- Architecture comparison — held-out results and bootstrap intervals
- Failure analysis — observed detector, quantity, billing, and two-stage failure cases
- Architecture comparison plot and classifier confusion matrix
- Portfolio upgrade report — deeper engineering and experiment narrative
The local Freiburg Groceries v10 metadata identifies the dataset as CC BY 4.0 and attributes its distribution to Roboflow. The validated demo image is from that dataset and is used under the same license. Dataset exports and generated crops are not committed to this repository.
The qualitative checkout demo is a curated, project-generated asset used only to demonstrate the checkout interface. It is excluded from model evaluation and threshold selection.