Gather batch-parallel validation payloads

[bghira]

Summary

• gather distributed validation payloads via Accelerate object gathering
• treat gathered validation payloads as a flat ordered list before rank 0 applies media results
• add regression coverage that batch-parallel audio validation saves peer-rank audio payloads

Root Cause

Batch-parallel validation split prompt work with the accelerator, but gathered result payloads through SimpleTuner's torch-distributed helper. In runs where that helper could not see an initialized torch-distributed group, rank 0 only received its own local payload list, so audio samples generated on peer ranks were never added to the saved validation audio set.

Validation

• .venv/bin/python -m unittest -v tests.test_validation_context_parallel tests.test_validation_audio_mock tests.test_validation_external

[Copilot]

Pull request overview

This PR updates distributed validation in Validation.process_prompts to gather batch-parallel validation payloads via accelerate.utils.gather_object, aiming to ensure rank 0 receives and persists media payloads (notably audio) produced on peer ranks.

Changes:

• Switched distributed validation payload gathering from the project’s torch-distributed helper to accelerate.utils.gather_object.
• Adjusted aggregation flow to treat gathered payloads as a single ordered collection before applying results on rank 0.
• Added a regression test asserting batch-parallel audio validation includes peer-rank audio payloads in the saved outputs.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File	Description
simpletuner/helpers/training/validation.py	Replaces cross-process payload gathering implementation used during distributed validation.
tests/test_validation_context_parallel.py	Adds regression coverage for gathering and saving audio payloads produced on non-zero ranks.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.