Skip to content

dataset_ingestion.md

Latest commit

 

History

History
217 lines (164 loc) · 4.68 KB

dataset_ingestion.md

File metadata and controls

217 lines (164 loc) · 4.68 KB
Raw

BDD100K Dataset Ingestion

The ingestion workflow converts BDD100K Detection 2020 annotations into a deterministic YOLO object-detection dataset.

Dataset files

BDD100K images and annotations are not included in this repository. Download them through the official distribution channel and follow the dataset license.

Place the files in:

data/raw/bdd100k/
|-- images/
|   `-- 100k/
|       |-- train/
|       `-- val/
`-- labels/
    `-- det_20/
        |-- det_train.json
        `-- det_val.json

Class schema

The canonical class mapping is stored in:

configs/class_schema.yaml

The ten canonical classes are:

0 person
1 rider
2 car
3 truck
4 bus
5 train
6 motorcycle
7 bicycle
8 traffic_light
9 traffic_sign

BDD100K Detection 2020 source categories are normalized as:

pedestrian    -> person
rider         -> rider
car           -> car
truck         -> truck
bus           -> bus
train         -> train
motorcycle    -> motorcycle
bicycle       -> bicycle
traffic light -> traffic_light
traffic sign  -> traffic_sign

The following ambiguous source categories are explicitly ignored:

other person
other vehicle
trailer

Any other unmapped source category remains a hard error.

Default pilot configuration

The default configuration is:

configs/dataset_ingestion.yaml

It selects 2,000 training images and 500 validation images using random seed 42. Source train and validation splits remain separate.

Annotation-balanced pilot

The balanced configuration is:

configs/dataset_ingestion_balanced.yaml

It preserves the official source splits, selects exactly 2,000 training images and 500 validation images, and targets at least 50 annotations per canonical class before seeded random filling.

Rare classes receive priority. Frames covering multiple unmet targets receive additional preference. Selection remains deterministic for the configured random seed.

The validation source split contains only 15 train annotations. The selector therefore records that unavoidable shortfall while selecting every available validation train annotation.

Verified output:

training images:       2000
validation images:      500
combined annotations: 45443
canonical classes:       10
combined train count:    66
imbalance ratio:     378.83

Generated files are isolated under:

data/interim/balanced/images/
data/interim/balanced/labels/
data/metadata/balanced_ingestion_manifest.csv

Run and validate the balanced pilot with:

edge-traffic-ingest --config configs/dataset_ingestion_balanced.yaml
edge-traffic-dataset-check --config configs/dataset_validation_balanced.yaml

Dry run

A dry run loads annotations, selects frames, decodes images, converts labels, validates destinations, and prepares the manifest without writing outputs.

edge-traffic-ingest `
    --config configs/dataset_ingestion.yaml `
    --dry-run

Execute ingestion

edge-traffic-ingest `
    --config configs/dataset_ingestion.yaml

Generated outputs:

data/interim/cleaned/images/train/
data/interim/cleaned/images/val/
data/interim/cleaned/labels/train/
data/interim/cleaned/labels/val/
data/metadata/ingestion_manifest.csv

Images without annotations receive empty YOLO label files.

Transaction behavior

Images, labels, and the CSV manifest are written as one transaction.

When writing fails:

  • Newly created outputs are removed
  • Existing overwritten files are restored
  • Empty directories created by the failed run are removed when possible

Existing outputs are rejected by default. Use explicit overwrite mode only when replacement is intended:

edge-traffic-ingest `
    --config configs/dataset_ingestion.yaml `
    --overwrite

Manifest fields

The manifest contains:

split
image_name
source_image_path
output_image_path
output_label_path
width
height
channels
annotation_count
class_ids
source_categories
image_transfer_mode

Validate generated dataset

edge-traffic-dataset-check `
    --config configs/dataset_validation_cleaned.yaml

Validation reports are written to:

reports/dataset_validation_cleaned/validation_report.json
reports/dataset_validation_cleaned/validation_issues.csv

Current limitations

  • Only BDD100K train and validation detection splits are supported
  • Annotation JSON files are loaded completely into memory
  • Detection labels must contain valid box2d coordinates
  • Out-of-bound boxes are rejected instead of clipped
  • Image transfer currently supports copy mode only
  • Source-level class imbalance remains, especially for the train class
  • The validation source split contains only 15 train annotations