Skip to content

Add manual system telemetry metrics logging - #3172

Merged
bghira merged 1 commit into
mainfrom
feature/3167-system-telemetry
Aug 31, 2026
Merged

Add manual system telemetry metrics logging#3172
bghira merged 1 commit into
mainfrom
feature/3167-system-telemetry

Conversation

@bghira

@bghira bghira commented Aug 31, 2026

Copy link
Copy Markdown
Owner

Summary

  • add a training-side system telemetry sampler for trackers without native host metrics
  • collect CPU, memory, output disk, network rate, and GPU metrics
  • support NVIDIA/CUDA, AMD/ROCm, and Apple/MPS telemetry sources where available
  • skip manual telemetry for WandB to avoid duplicating its native host metrics

Refs #3167.

Validation

  • .venv/bin/python -m unittest tests.test_system_metrics -v -f
  • .venv/bin/python -m py_compile simpletuner/helpers/training/system_metrics.py simpletuner/helpers/training/trainer.py
  • git diff --check

@bghira
bghira merged commit 90fbf3c into main Aug 31, 2026
4 checks passed
@bghira
bghira deleted the feature/3167-system-telemetry branch August 31, 2026 00:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant