Skip to content

low disk space detection and script execution action - #2460

Merged
bghira merged 1 commit into
mainfrom
feature/detect-low-disk-space
Jan 20, 2026
Merged

low disk space detection and script execution action#2460
bghira merged 1 commit into
mainfrom
feature/detect-low-disk-space

Conversation

@bghira

@bghira bghira commented Jan 20, 2026

Copy link
Copy Markdown
Owner

Closes #2353

This pull request introduces a comprehensive disk space monitoring and protection feature for checkpoint saves in SimpleTuner. It adds new configuration options for enforcing a minimum free disk space threshold, specifying actions to take when space is low (stop, wait, or run a cleanup script), and validates these settings both in the CLI and the UI. The feature is documented in multiple languages, and the codebase is updated to check disk space before saving checkpoints, ensuring more robust and user-friendly training runs.

Disk Space Monitoring and Protection

  • Added new configuration options:
    • disk_low_threshold: Minimum free disk space required before checkpoint saves.
    • disk_low_action: Action to take when space is low (stop, wait, or script).
    • disk_low_script: Path to a cleanup script for automated space recovery. [1] [2]
  • Implemented disk space checking in the checkpointing process:
    • Before saving a checkpoint, the trainer checks available disk space and takes the configured action if below threshold (stop, wait, or run script). [1] [2]
  • Added robust CLI and config validation:
    • Validates disk space options, ensures script path exists when required, and parses human-readable size strings. [1] [2]

Documentation Updates

  • Documented the new disk space options and behaviors in all supported languages:
    • English, Spanish, Portuguese (Brazil), Hindi, Japanese, and Chinese documentation files updated with clear explanations and usage examples. [1] [2] [3] [4] [5] [6]

@bghira
bghira merged commit 329073d into main Jan 20, 2026
4 checks passed
@bghira
bghira deleted the feature/detect-low-disk-space branch January 20, 2026 19:12
samzhang996-hue pushed a commit to samzhang996-hue/SimpleTuner that referenced this pull request Apr 27, 2026
low disk space detection and script execution action
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[feature request] add mode to pause instead of stop when running out of space

1 participant