Skip to content

TESTING_GUIDE.md

Latest commit

 

History

History
349 lines (270 loc) · 10.2 KB

TESTING_GUIDE.md

File metadata and controls

349 lines (270 loc) · 10.2 KB
Raw

Testing Guide

This guide explains how to use the test utilities to evaluate crop quality on sample images.

Test Utilities

1. Generate Test Set (scripts/generate_test_set.py)

Creates a random sample of images from a source directory for testing.

Usage:

python scripts/generate_test_set.py \
  --source ~/stuff/onedrive-album-download/downloads/art/ \
  --output test_real_images/input \
  --count 64 \
  --seed 42

Options:

  • --source, -s: Source directory containing images (required)
  • --output, -o: Output directory for test set (required)
  • --count, -n: Number of images to select (default: 64)
  • --seed: Random seed for reproducibility (optional)

Use Cases:

  • Create representative test sets for quality assessment
  • Reproducible test sets with seed parameter
  • Quick sampling from large image collections

2. Generate Interactive Detection Report (src/frame_prep/report.py)

Creates a comprehensive HTML report showing detection results alongside cropped outputs for quality assessment.

Usage:

frame-prep report

Features:

  • Configuration Summary: Shows all detection and cropping parameters at the top
    • Detection Strategy: Models used (YOLO-World + Grounding DINO)
    • Detection Parameters: Confidence threshold, merge threshold
    • Cropping Strategy: Target dimensions, zoom factor, fallback settings
  • Side-by-side Comparison: Each image shows:
    • Detection view (left): Original image with bounding boxes
    • Result view (right): Actual cropped output with zoom factor
  • Ground Truth Comparison: Compares detections against annotated ground truth
  • Accuracy Metrics: IoU scores and overall accuracy percentage
  • Feedback System: Rate results as Good/Poor/Zoom Issue with comments
  • Export: Save feedback as JSON for analysis

Output: reports/interactive_detection_report.html

Configuration Parameters Displayed:

Section Parameter Description
Detection Ensemble OptimizedEnsembleDetector
Detection Models YOLO-World (yolov8m-worldv2) + Grounding DINO (tiny)
Detection Confidence Threshold Minimum detection confidence (default: 0.25)
Detection Merge Threshold IoU threshold for merging boxes (default: 0.2)
Cropping Target Dimensions Output size (default: 480×800)
Cropping Max Zoom Factor Maximum zoom cap (default: 8.0x)
Cropping Saliency Fallback Use saliency when no detections (default: enabled)

3. Interactive Detection Report (frame-prep report)

The primary quality assessment tool. Creates a comprehensive HTML report with detection overlays and crop results.

Usage:

frame-prep report

Output: reports/interactive_detection_report.html

Features:

  • Side-by-side detection view (bounding boxes) and crop result
  • Ground truth comparison with IoU scores
  • Rate results as Good/Poor/Zoom Issue with comments
  • Export feedback as JSON for analysis
  • Multi-crop display for images with multiple detected subjects

Complete Workflow

Step 1: Generate Test Set

# Create 64-image random sample
python scripts/generate_test_set.py \
  --source ~/stuff/onedrive-album-download/downloads/art/ \
  --output test_real_images/input \
  --count 64 \
  --seed 42

Step 2: Process Images

# Process with smart strategy (default)
frame-prep batch \
  --input-dir test_real_images/input/ \
  --output-dir test_real_images/output/ \
  --width 480 --height 800 \
  --workers 4

Step 3: Generate Report

frame-prep report

Step 4: Review and Rate

  1. Open reports/interactive_detection_report.html in your browser
  2. Review each image comparison
  3. Rate crop quality (1-5 stars)
  4. Add comments for problematic crops
  5. Export feedback as JSON

Step 5: Analyze Results

The exported JSON contains:

{
  "timestamp": "2026-01-30T...",
  "totalImages": 64,
  "ratedImages": 64,
  "feedback": {
    "1": {
      "rating": 5,
      "filename": "image1.jpg",
      "comment": "Perfect crop"
    },
    "2": {
      "rating": 3,
      "filename": "image2.jpg",
      "comment": "Subject slightly cut off"
    }
  }
}

Quality Rating Guidelines

⭐⭐⭐⭐⭐ Excellent (5 stars)

  • All important subjects preserved
  • Good composition
  • Nothing important cut off
  • Visually appealing crop

⭐⭐⭐⭐ Good (4 stars)

  • Main subjects preserved
  • Minor elements may be cropped
  • Overall satisfactory result

⭐⭐⭐ Fair (3 stars)

  • Main subject partially preserved
  • Some important elements cropped
  • Acceptable but not ideal

⭐⭐ Poor (2 stars)

  • Important subjects cut off
  • Poor composition
  • Significant quality loss

⭐ Bad (1 star)

  • Critical subjects missing
  • Unusable crop
  • Wrong area selected

Analyzing Feedback

Calculate Statistics

import json

with open('crop_quality_feedback_2026-01-30.json') as f:
    data = json.load(f)

ratings = [f['rating'] for f in data['feedback'].values() if 'rating' in f]

print(f"Total rated: {len(ratings)}")
print(f"Average: {sum(ratings)/len(ratings):.2f}")
print(f"Excellent (5): {ratings.count(5)}")
print(f"Good (4): {ratings.count(4)}")
print(f"Fair (3): {ratings.count(3)}")
print(f"Poor (2): {ratings.count(2)}")
print(f"Bad (1): {ratings.count(1)}")

Find Problematic Crops

# Images rated 2 or below
problematic = [
    (id, f['filename'], f.get('comment', ''))
    for id, f in data['feedback'].items()
    if f.get('rating', 0) <= 2
]

for img_id, filename, comment in problematic:
    print(f"{filename}: {comment}")

Testing Different Strategies

Compare different cropping strategies:

# Smart strategy (default)
frame-prep batch \
  -i test_real_images/input/ \
  -o test_real_images/smart/ \
  --strategy smart

# Saliency strategy
frame-prep batch \
  -i test_real_images/input/ \
  -o test_real_images/saliency/ \
  --strategy saliency

# Center strategy
frame-prep batch \
  -i test_real_images/input/ \
  -o test_real_images/center/ \
  --strategy center

# Generate report for each
frame-prep report

Reproducible Testing

Use the same seed for reproducible test sets:

# Everyone gets the same 64 images
python scripts/generate_test_set.py \
  --source ~/images/ \
  --output test_set/ \
  --count 64 \
  --seed 42

Tips

  1. Start small: Test with 8-16 images first
  2. Increase gradually: Move to 64 for comprehensive testing
  3. Use seeds: Reproducible results for comparisons
  4. Export regularly: Save feedback JSON periodically
  5. Note patterns: Look for systematic issues in comments
  6. Test edge cases: Include various image types (portraits, landscapes, abstract)

Automation

Create a testing script:

#!/bin/bash
# test_pipeline.sh

# Generate test set
python scripts/generate_test_set.py \
  -s ~/art/ -o test/input -n 64 --seed 42

# Process images
frame-prep batch \
  -i test/input -o test/output

# Generate report
frame-prep report

# Open in browser
xdg-open reports/interactive_detection_report.html

Make it executable:

chmod +x test_pipeline.sh
./test_pipeline.sh

Feedback Analysis History

Configuration Comparison (2026-02-02)

Configuration Accuracy Bad Count Processing Time Notes
yolov8m + single-pass ~85% 8-10 Fast (baseline) Current default
yolov8l + single-pass 86.9% 7 ~40% slower Marginal improvement
yolov8l + two-pass (60% center) 93.75% 4 ~3-4x slower Best accuracy but too slow
Lower conf (0.20) 85.7% 8 Same More detections but no net gain

Conclusion: Two-pass detection with center crop significantly improves accuracy (93.75% vs ~85%) but the processing time increase (~3-4x) is not worth the tradeoff for typical use cases. The two-pass code is retained but disabled by default.

Two-pass detection notes:

  • Crops center 60% of image and runs detection again
  • Maps coordinates back to original image space
  • Helps detect small central subjects that get lost in full-image context
  • Improves both detection AND labeling (objects are larger relative to frame)

2026-02-02 Feedback Summary (Latest)

Accuracy: 82.8% (53/64 good) with YOLO-World class name mapping enabled.

Key Insight: Most failures are prioritization issues, not detection failures. The correct art pieces are detected but wrong ones are selected as primary.

Current Configuration:

  • Confidence threshold: 0.25
  • Max zoom factor: 8.0x (aggressive for tiny subjects)
  • Primary subject selection: Center-weighted with class priorities and size scoring
  • YOLO-World returns actual class names for proper prioritization

Prioritization Failures (from feedback):

Image Issue Correct Detection Exists
#3 Collage ("sculpture statue") not prioritized over other detections Yes
#11 Full-height mural ("painted figure", "painting") not selected Yes
#18 Central "painted figure" or "art installation street art" missed Yes
#53 "sculpture statue figurine" in center lost to street lamp on right Yes

Actionable Improvements:

  1. Add to avoid_classes: 'street lamp', 'lamp post', 'light pole'
  2. Boost priority for: 'painted figure', 'mural', 'art installation'
  3. Increase center-weighting: Central detections should beat edge detections more decisively
  4. Multi-class detection: YOLO-World may return multiple classes per box - investigate using all matches

Classes Removed (caused false positives):

  • graffiti - triggered on plain walls
  • stencil - too generic
  • bird statue - confused with other objects

Successful Tuning Changes:

  1. YOLO-World now returns actual class names (not yolo:X IDs) enabling proper prioritization
  2. Size scoring with aggressive penalties for tiny objects (< 1% of frame)
  3. Center-weighting to prefer subjects near image center
  4. Class priority system distinguishing high-value art from noise

Future Improvement Ideas:

  1. Strengthen center-weighting to more decisively prefer central subjects
  2. Add "collage" to high-priority art classes
  3. Investigate YOLO-World multi-class scores per detection
  4. Consider training a custom model on art-specific dataset
  5. Implement multi-subject mode for images with multiple art pieces