[feat] GenRL: fix PPO loop cadence and diagnostics

[Abecid]

Extracted from #1391.

GenRL-Stack: 4/6

Purpose

Fix GenRL PPO/GRPO training-loop behavior and diagnostics without mixing in reward-model or LoRA changes.

This also addresses review feedback from Gemini/David on sampler validation, accumulated microbatch gradients, and post-update KL diagnostics.

Fixes #

Changes

• Add GenRL config validation for unsupported reward/reference/loss-reweighting combinations.
• Refactor PPO loss computation into a helper so normal and accumulated paths share the same objective.
• Add accumulated rollout microbatch training support.
• Support optimizer stepping per timestep or after all train timesteps.
• Fix accumulated microbatch zero_grad() when optimizer_step_per_timestep=false.
• Refresh post-update probe args per timestep instead of always probing timestep 0.
• Log post-update KL diagnostics in the non-accumulated, non-per-timestep optimizer path.
• Add trainable-parameter validation/logging.
• Hook optional evaluation metrics into GenRL step metrics.

Test Plan

python -m py_compile fastvideo/train/methods/rl/genrl.py

Test Results

Test output

py_compile passed locally.

[mergify]

⚠️ PR title format required

Your PR title must start with a type tag in brackets. Examples:

• [feat] Add new model support
• [bugfix] Fix VAE tiling corruption
• [refactor] Restructure training pipeline
• [perf] Optimize attention kernel
• [ci] Update test infrastructure
• [infra] Add activation trace hooks
• [docs] Add inference guide
• [misc] Clean up configs
• [new-model] Port Flux2 to FastVideo
• [skill] Add add-model agent skill

Valid tags: feat, feature, bugfix, fix, refactor, perf, ci, infra, doc, docs, misc, chore, kernel, new-model, skill, skills

Please update your PR title and the merge protection check will pass automatically.

[gemini-code-assist]

Code Review

This pull request introduces several enhancements to the GenRL training pipeline, including configuration validation, periodic evaluation during training, logging of trainable parameters, and support for PPO microbatch accumulation. However, the reviewer identified critical issues that must be addressed: an ImportError caused by importing the undefined function clear_reward_models, a potential TypeError when calling eval_once with unsupported arguments (max_batches and seed), and a potential shape mismatch crash when slicing self._train_neg_embeds during microbatch accumulation.