Skip to content

Commit f795285

Browse files
committed
llm_post_training: Clarify that PPO ratio clip is inactive
1 parent 7bf4fbd commit f795285

1 file changed

Lines changed: 1 addition & 1 deletion

File tree

‎llm_post_training/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -659,13 +659,13 @@ The recipe uses token-level GRPO with:
659659
- a leave-one-out baseline;
660660
- no reward shaping;
661661
- no reference-policy KL penalty (`reference_policy_kl_penalty: 0`);
662-
- asymmetric PPO ratio clipping configured as `0.2` and `0.28`;
663662
- asynchronous collection, with the checked-in RCPs using a maximum trajectory
664663
age of one training step;
665664
- sequence-mask truncated importance sampling (`seq-mask-tis`) with lower and
666665
upper bounds `0.999` and `1.002`;
667666
- importance-sampling correction enabled;
668667
- `force_on_policy_ratio: true`;
668+
- asymmetric PPO ratio clipping is not used due to `force_on_policy_ratio: true` (any configured values are ignored)
669669
- reference-policy log-probability calculation skipped;
670670
- sequence-level log-probability error masking with threshold `2.0`; and
671671
- overlong-response filtering enabled.

0 commit comments

Comments
 (0)