storage: percentage / free-space based retention (min_free_space)

[fluffyspace]

Summary

Adds an optional min_free_space retention rule to the storage component so a tier can be told to "keep at least X% (or X GB) of the filesystem free", evaluated live, instead of (or in addition to) the existing absolute per-camera max_size byte cap.

Motivation

Today a tier's size limit is only expressible as absolute bytes, enforced per camera per tier (max_size: { gb, mb }). That has two sharp edges on real deployments:

1. It doesn't compose with the number of cameras. Operators reason about max_size as a total, but the true worst case is per-camera cap × number of cameras. On a shared disk it's easy to under-provision and fill the volume.
2. It's blind to the rest of the disk. When the tier's path shares a filesystem with other data, a byte cap can't express "always leave headroom on the volume" — which is usually what you actually want on a shared disk.

What this does

storage:
  recorder:
    tiers:
      - path: /recordings
        continuous:
          max_age: { days: 7 }
          min_free_space: { percent: 15 }   # or { gb: 40 }

When set, the oldest files on that tier across all cameras on the filesystem are evicted whenever live free space (shutil.disk_usage) drops below the floor. It layers on top of max_age/max_size — a file is removed when any configured rule fires. Accepted on the recorder continuous/events blocks and on snapshot/timelapse tiers.

Fully backward compatible: unset ⇒ floor 0 ⇒ the previous size/age-only behaviour, exactly. No DB/schema migration.

Implementation

Free space is a filesystem-global property while the existing tier check is per-camera, so this is a dedicated all-cameras eviction pass rather than a change to the per-camera path:

• util.calculate_free_space_floor — resolves percent-of-total + absolute gb/mb to a byte floor (the largest wins); total read once via disk_usage.
• check_tier.get_files_to_delete_for_free_space — pure-numpy oldest-first selection covering the deficit, with a write-protection window guarding freshly written segments (active HLS).
• check_tier.load_free_space_candidates — loads a tier's files across all cameras for one subcategory.
• check_tier.Worker.free_space_evict — piggybacked on each camera's check_tier, but serialized + throttled per filesystem so N concurrent camera checks cause one eviction pass, not N. Reuses the existing delete_file path.
• DataItem carries min_free_bytes + tier_fs_path (defaults keep it inert); tier handlers resolve the floor once at init and plumb it through.

Design notes

• Deletion, never move — only deletion returns space to a disk, so this is meant for the terminal tier of a filesystem.
• btrfs-safe — selection is a single size-summed batch (no per-row disk_usage re-poll), so a filesystem whose accounting lags behind unlinks isn't over-evicted.
• Known limitation — under genuine disk pressure this can evict old continuous segments that belong to an event recording (that recording's playback then 404s). It's a last-resort backstop; happy to add orphan-only (recording-aware) protection if you'd prefer that as the default.

Tests

• test_free_space.py — compute selection (oldest-first, exact boundaries, min-age protection, empty/no-deficit).
• test_util.py — calculate_free_space_floor (percent / absolute / max-of-several / stat-failure / unset).
• test_config.py — schema accept/reject for min_free_space (percent (0,100]) + defaults presence on continuous/events/snapshots.

[netlify]

✅ Deploy Preview for viseron canceled.

Name	Link
🔨 Latest commit	ee8fe37
🔍 Latest deploy log	https://app.netlify.com/projects/viseron/deploys/6a65169c70ef9e0008cf3e7b

[roflcoopter]

Thank you, this seems like a great addition. I am preparing for a new release atm, will look into this when that is done

[roflcoopter]

This description is a little too chatty, try to shorten it if possible

[roflcoopter]

Loading all files for a tier could potentially cause memory issues, maybe this one would suite better as purely an SQL query instead of loading and using numpy.

What do you think about that?

The reason why the normal eviction uses loading + numpy is because a similar SQL query became extremely complicated and slow, using numpy was the better choice there to get better control of the CPU usage using throttling etc