VisionCart began as a 17-cell Colab notebook coupling Roboflow, Drive, camera JavaScript, YOLO, ResNet, Google Sheets, float billing, logging, and voice. It is now a local, tested ML application with reproducible data preparation, held-out evaluation, confidence calibration, Decimal billing, service abstractions, one-command orchestration, and evidence-backed architecture selection.
The final production path is YOLOv8s-only. On 494 untouched test scenes it achieved 0.8363 item F1, 57.09% exact-basket accuracy, 60.53% quantity accuracy, and $1.69 billing MAE. It beat YOLO + ResNet-50 on every checkout metric and ran 116.95 ms faster at median CPU latency.
The exact Freiburg Groceries v10 YOLOv8 export was acquired from Roboflow. Dataset metadata identifies 25 classes and CC BY 4.0.
| Split | Images | Boxes | Ground-truth classifier crops |
|---|---|---|---|
| Train | 3,946 | 9,231 | 9,231 |
| Validation | 493 | 1,178 | 1,178 |
| Test | 494 | 1,199 | 1,199 |
The full validator found no corrupt images, missing annotations, malformed labels, invalid coordinates, invalid class IDs, empty annotations, missing classes, skipped crops, or catalog mismatches. SHA-256 source comparison found zero identical images across train, validation, and test. Every crop manifest entry retains source path, source hash, split, annotation index, class, and output path.
The recovered detector is an 11.1M-parameter fine-tuned pretrained YOLOv8s checkpoint with the exact dataset class ordering. It was retained because its held-out results are strong and valid; retraining the detector on CPU offered no evidence-backed benefit over this established baseline.
| Test metric | Result |
|---|---|
| Precision | 0.8369 |
| Recall | 0.8379 |
| F1 | 0.8374 |
| mAP@50 | 0.9038 |
| mAP@50:95 | 0.7631 |
Classifier crops use ground-truth boxes and IDs, correcting the notebook's prediction-teacher methodology. The historical and newly reproducible models were evaluated on the same 1,199 test crops after all training decisions.
| Checkpoint | Top-1 | Top-5 | Macro F1 |
|---|---|---|---|
| Historical inference-only | 76.31% | 95.08% | 0.7391 |
| New reproducible epoch 0 | 64.80% | 89.66% | 0.6203 |
The new checkpoint stores model, optimizer, scheduler, epoch, class names, validation score, and configuration. A resume-only continuation restored epoch 0 and its 0.6418 validation score through the normal training path. Full-network CPU fine-tuning took approximately 19 minutes per epoch; the one completed epoch provides a genuine reproducibility artifact, while the stronger historical model was honestly retained for the architecture experiment.
The strongest historical classes by F1 were corn, water, juice, sugar, and tomato sauce. The weakest were chips, oil, nuts, chocolate, and rice. Oil → vinegar was the largest confusion (17), followed by chocolate → coffee, tea → coffee, and water → soda (8 each).
Detector thresholds 0.10–0.50 and classifier review thresholds 0.00–0.80 were compared on 493 validation scenes. Selection prioritized exact baskets, item F1, exact billing, billing MAE, and quantity accuracy. The frozen defaults are:
- detector confidence: 0.50;
- classifier review confidence: 0.50;
- NMS IoU: 0.35;
- final matching IoU: 0.50.
At the chosen two-stage threshold, validation accepted coverage was 87.20% and review rate was 12.80%. No threshold was optimized on test data.
Both paths used the identical 494 test scenes, static prices, matching logic, and frozen validation-selected thresholds.
| Metric | YOLO-only | YOLO + ResNet-50 |
|---|---|---|
| True positives | 1,019 | 827 |
| False positives | 219 | 284 |
| False negatives | 180 | 372 |
| Item precision | 0.8231 | 0.7444 |
| Item recall | 0.8499 | 0.6897 |
| Item F1 | 0.8363 | 0.7160 |
| Exact basket accuracy | 57.09% | 42.71% |
| Quantity accuracy | 60.53% | 55.26% |
| Billing MAE | $1.69 | $2.29 |
| Exact billing rate | 57.09% | 42.91% |
| Median CPU latency | 150.48 ms | 267.43 ms |
| Approximate FPS | 6.72 | 3.78 |
Two thousand scene-level bootstrap resamples gave YOLO-only a 95% item-F1 interval of 0.8115–0.8575 versus 0.6826–0.7464 for two-stage. Exact-basket intervals were 52.83%–61.54% and 38.46%–47.17%, respectively.
YOLO-only became the default because it improved exact baskets by 14.37 percentage points, item
F1 by 0.1203, quantity accuracy by 5.26 points, exact billing by 14.17 points, and billing MAE by
$0.61 while reducing median latency 43.73%. ResNet was not retained merely for architectural
complexity; it is available behind --two-stage for continued experimentation.
Windows 11, Intel Core i7-12700H-class CPU, 20 logical CPUs, 15.7 GB RAM, PyTorch 2.9.1 CPU; 20 held-out 256×256 images, 3 warmups, and 20 timed repetitions per image:
| Stage | Median | Mean | p95 |
|---|---|---|---|
| YOLO-only detector | 148.54 ms | 149.07 ms | 158.68 ms |
| YOLO-only total | 150.48 ms | 150.66 ms | 160.96 ms |
| ResNet preprocessing/crop | 1.52 ms | 1.53 ms | 2.03 ms |
| ResNet inference/crop | 53.35 ms | 53.27 ms | 58.28 ms |
| Two-stage total | 267.43 ms | 292.76 ms | 409.92 ms |
Held-out artifacts document all requested case types:
- Beans were sometimes confused with tomato sauce, jam, or nuts, causing simultaneous FP, FN, quantity, and billing errors.
- One five-bean scene produced 0 TP, 3 FP, and 5 FN; its expected $7.50 basket was billed $8.47.
- ResNet broke correct YOLO predictions in a six-bean scene, contributing to $31.44 predicted versus $9.00 ground-truth value.
- ResNet also corrected a YOLO error on another beans image, showing that its net harm comes from inconsistency rather than an inability to add value anywhere.
- Overlap caused one miss in a four-candy scene, while a small-object seven-candy scene detected only three items.
- Low-confidence classifier review was observed and rendered instead of silently forced.
- Local image and freeze-confirmed webcam checkout replaced Colab-only execution.
- Detector and classifier confidences are distinct; review blocks unresolved checkout items.
- Decimal billing rejects missing prices instead of silently billing zero.
- Local CSV/JSONL providers make Google integrations optional and lazy.
- Dataset download reads process environment first and private root
.envsecond usingpython-dotenvwithoverride=False. .envis ignored;.env.examplecontains a blank key; scans skip private.envcontents while verifying that the file is ignored and scanning committed/template/report/notebook content.- The one-command pipeline validates data, checks leakage, creates crops, reuses or trains models, tunes validation thresholds, evaluates both architectures, benchmarks, generates failures, checks configuration against the measured winner, and refreshes reports.
Checkout requires both identity and quantity-producing localization. The recovered YOLOv8s offers a strong accuracy/latency balance: 0.9038 mAP@50 and 0.7631 mAP@50:95 on test.
It tested whether focused crop classification could improve visually similar packaging. The experiment was falsifiable and same-split; it reduced end-to-end accuracy and added latency, so it was removed from the default path.
Prediction-labeled crops teach the classifier to imitate detector mistakes. Ground-truth boxes and class IDs provide independent supervision, while source hashes prove split isolation.
Item F1 averages errors across objects, while checkout succeeds only when every product and quantity in a scene is correct. Exact-basket accuracy therefore captures customer-visible transaction correctness.
Thresholds were selected on validation only. The optional classifier sends predictions below 0.50 to review and blocks billing rather than assigning a potentially wrong SKU.
Collect checkout-camera data emphasizing beans, chips, oil/vinegar, chocolate/coffee, overlap, small objects, glare, and packaging changes; then test calibrated unknown-product rejection and detector error-directed fine-tuning.
VisionCart — AI-Powered Autonomous Checkout
Python, PyTorch, YOLOv8s, ResNet-50, OpenCV
- Engineered a 25-class visual checkout system spanning YOLOv8s detection, confidence validation, dynamic pricing, Decimal billing, receipt generation, and structured transaction logging; achieved 0.9038 mAP@50 and 0.7631 mAP@50:95 on 494 held-out scenes.
- Built leakage-safe ground-truth crop generation and reproducible training/evaluation pipelines; delivered 0.8363 end-to-end item F1, 57.09% exact-basket accuracy, and $1.69 billing MAE on the untouched test split.
- Benchmarked YOLO-only against YOLO + ResNet-50 and selected YOLO-only after it improved exact baskets by 14.37 points while reducing median CPU latency from 267.43 ms to 150.48 ms.
- Exact-basket accuracy of 57.09% is not sufficient for unattended retail deployment.
- The model is closed-set and requires product/packaging refresh data.
- The reproducible ResNet checkpoint received one full CPU epoch; the historical inference model remains stronger but lacks original optimizer history.
- Webcam hardware, Google Sheets, and speech depend on external devices/services and are optional.
- Continuous tracking, payment, and inventory integration remain outside project scope.