Repository navigation
Expand file tree
/
Copy pathreader.html
More file actions
2383 lines (2320 loc) · 154 KB
/
Copy pathreader.html
File metadata and controls
2383 lines (2320 loc) · 154 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>Read - RL without Tears</title>
<link rel="icon" type="image/png" sizes="32x32" href="assets/favicon/favicon-32.png">
<link rel="apple-touch-icon" sizes="180x180" href="assets/favicon/apple-touch-icon.png">
<link rel="stylesheet" href="styles.css?v=20260911">
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/katex@0.16.11/dist/katex.min.css">
<script defer src="https://cdn.jsdelivr.net/npm/katex@0.16.11/dist/katex.min.js"></script>
<script defer src="https://cdn.jsdelivr.net/npm/katex@0.16.11/dist/contrib/auto-render.min.js"></script>
<script defer src="math.js"></script>
</head>
<body>
<header class="topbar">
<a class="brand" href="index.html">RL without Tears</a>
<nav class="topnav" aria-label="Reader navigation">
<a href="#introduction">Introduction</a>
<a href="#preliminaries">Preliminary</a>
<a href="#rl-training">Understanding RL</a>
<a href="#improved-rl">Improved RL</a>
<a href="#reasoning">Reasoning</a>
<a href="#agents">Agents</a>
<a href="#multimodal">Multimodal</a>
<a href="#conclusion">Future</a>
<a href="cheatsheet.html">Cheat Sheet</a>
<a href="glossary.html">Glossary</a>
</nav>
</header>
<main class="layout">
<aside class="sidebar" aria-label="Chapter navigation">
<nav class="chapter-nav">
<details open>
<summary><a href="#introduction">1. Introduction</a></summary>
<a class="nav-sub" href="#intro-motivation">Why RL for LLMs?</a>
<a class="nav-sub" href="#intro-gap">The Gap This Paper Fills</a>
<a class="nav-sub" href="#intro-roadmap">Roadmap</a>
</details>
<details>
<summary><a href="#preliminaries">2. Preliminary</a></summary>
<a class="nav-sub" href="#llm-fundamentals">2.1 LLM Fundamentals</a>
<a class="nav-sub" href="#rl-fundamentals">2.2 RL Fundamentals</a>
</details>
<details>
<summary><a href="#rl-training">3. Understanding RL</a></summary>
<a class="nav-sub" href="#policy-gradient">3.1 Policy Gradient</a>
<a class="nav-sub" href="#temporal-decomposition">3.2 Temporal Decomposition</a>
<a class="nav-sub" href="#variance-reduction">3.3 Variance Reduction</a>
<a class="nav-sub" href="#importance-sampling">3.4 Importance Sampling</a>
<a class="nav-sub" href="#ppo">3.5 PPO</a>
<a class="nav-sub" href="#reward-models">3.6 Reward Models</a>
<a class="nav-sub" href="#complete-workflow">3.7 Workflow</a>
</details>
<details>
<summary><a href="#improved-rl">4. Improved RL</a></summary>
<a class="nav-sub" href="#advanced-reward-models">4.1 Advanced Reward Models</a>
<a class="nav-sub" href="#better-advantage">4.2 Advantage Estimation</a>
<a class="nav-sub" href="#efficient-rl">4.3 Efficient RL</a>
<a class="nav-sub" href="#dpo-section">4.4 DPO</a>
</details>
<details>
<summary><a href="#reasoning">5. Reasoning</a></summary>
<a class="nav-sub" href="#test-time-scaling">5.1 Test-time Scaling</a>
<a class="nav-sub" href="#iterative-rl">5.2 Iterative RL</a>
<a class="nav-sub" href="#large-scale-rl">5.3 Large-scale RL</a>
<a class="nav-sub" href="#opd-section">5.4 OPD</a>
</details>
<details>
<summary><a href="#agents">6. Agents</a></summary>
<a class="nav-sub" href="#agent-capabilities">6.1 Capabilities</a>
<a class="nav-sub" href="#environment-design">6.2 Environments</a>
<a class="nav-sub" href="#credit-assignment">6.3 Credit Assignment</a>
<a class="nav-sub" href="#agent-experience">6.4 Learning from Experience</a>
</details>
<details>
<summary><a href="#multimodal">7. Multimodal</a></summary>
<a class="nav-sub" href="#multimodal-understanding">7.1 Understanding</a>
<a class="nav-sub" href="#multimodal-generation">7.2 Generation</a>
</details>
<details>
<summary><a href="#conclusion">8. Future Directions</a></summary>
<a class="nav-sub" href="#future-pretraining">RL for Pre-training</a>
<a class="nav-sub" href="#future-efficiency">Efficient RL</a>
<a class="nav-sub" href="#future-scaling">Scaling Limits</a>
<a class="nav-sub" href="#future-continual">Self-evolving RL</a>
</details>
<a class="nav-section" href="#systems">Appendix</a>
<a class="nav-section" href="#references">References</a>
<a class="nav-section" href="assets/rl-without-tears.pdf">Full PDF</a>
</nav>
</aside>
<article class="content reader-content">
<section class="reader-title">
<p class="kicker">Online reader</p>
<h1>RL without Tears</h1>
<p>
The online version follows the chapter structure of the paper, but it is written as a guided reading
path rather than a compressed abstract. Each section answers the question left by the previous one:
what an LLM policy is, how rewards can train it, why PPO-style RLHF needs stabilizers, and how the same
feedback loop later extends to reasoning, agents, and multimodal models. The PDF remains the complete
version for derivations, references, and detailed discussion.
</p>
<div class="lesson-map" aria-label="Reading path">
<a href="#introduction">1. Motivation</a>
<a href="#preliminaries">2. Connect token generation to RL notation</a>
<a href="#rl-training">3. Build PPO-style RLHF step by step</a>
<a href="#improved-rl">4. Fix the weak points of the basic workflow</a>
<a href="#reasoning">5. Turn verifiable reasoning into training signal</a>
<a href="#agents">6. Extend RL to interactive agents</a>
<a href="#multimodal">7. Apply the policy view beyond text</a>
<a href="#conclusion">8. Future directions</a>
</div>
</section>
<section class="chapter" id="introduction">
<p class="chapter-number">Section 1</p>
<h2>Introduction</h2>
<p>
Reinforcement learning is a framework for improving behavior from feedback. An agent makes decisions,
receives signals from an environment, and adjusts its policy so that future decisions collect higher
reward. Traditional introductions often begin with games, robotics, or control. This tutorial begins
instead with LLMs: a model receives a prompt, generates a response token by token, and can be evaluated
after the response or interaction is complete <a class="citation" href="#ref-1">[1]</a>.
</p>
<h3 id="intro-motivation">Why RL for LLMs?</h3>
<p>
The paper begins from a simple mismatch. Pre-training and supervised fine-tuning teach an LLM by showing
it text to imitate, but many things we want from an assistant are easier to judge than to demonstrate
exhaustively. Helpfulness, preference alignment, safety, reasoning correctness, and task success are
often properties of a complete behavior, not properties of one isolated token. RLHF uses feedback over
generated responses to align pretrained models with human preferences, while recent reasoning and agent
systems extend the same idea toward learning from verifiers, tools, environments, and accumulated
experience <a class="citation" href="#ref-3">[3]</a><a class="citation" href="#ref-2">[2]</a><a class="citation" href="#ref-12">[12]</a>.
</p>
<aside class="callout">
<strong>Motivation.</strong>
LLM post-training increasingly depends on behavior-level feedback: the model generates candidate
behavior, the behavior is evaluated, and optimization makes better behavior more likely.
</aside>
<figure class="quote-block">
<blockquote>
Intelligence, and its associated abilities, can be understood as subserving the maximisation of reward
by an agent acting in its environment.
</blockquote>
<figcaption>
- Silver, Singh, Precup, and Sutton <a class="citation" href="#ref-24">[24]</a>
</figcaption>
</figure>
<h3 id="intro-gap">The Gap This Paper Fills</h3>
<p>
For NLP researchers, the difficulty is rarely the motivation. The difficulty is the translation layer.
RL papers talk about states, actions, policies, trajectories, returns, critics, and advantages; LLM
papers talk about prompts, tokens, log probabilities, completions, preference data, reward models, and
KL penalties. The two vocabularies describe compatible objects, but they are often introduced from
different examples. This tutorial fills that gap by keeping the mathematical language of RL while
grounding each concept in an LLM training situation.
</p>
<table class="concept-table compact">
<thead>
<tr>
<th>Reader Question</th>
<th>How the Paper Answers It</th>
</tr>
</thead>
<tbody>
<tr><td>How is an LLM a policy?</td><td>Map the prompt and generated prefix to a state, and the next token to an action.</td></tr>
<tr><td>Why use rewards instead of only demonstrations?</td><td>Rewards evaluate generated behavior when exact target outputs are hard to specify.</td></tr>
<tr><td>Why are value functions and advantages needed?</td><td>They reduce gradient variance and make policy optimization more stable.</td></tr>
<tr><td>Why do PPO, GRPO, and DPO matter?</td><td>They represent different ways to optimize policies from reward or preference feedback.</td></tr>
</tbody>
</table>
<h3 id="intro-roadmap">Roadmap</h3>
<p>
The introduction then sets the scope of the paper. The tutorial proceeds in the order a reader needs the
concepts. Section 2 first builds the shared notation: LLM generation is a sequence of decisions, and RL
gives names to the decision process and feedback. Section 3 then turns that notation into a complete
PPO-style RLHF workflow. Section 4 revisits the workflow and asks where it breaks or becomes expensive.
Sections 5 to 7 apply the same lens to reasoning, agents, and multimodal models, where trajectories
become longer, rewards become more varied, and credit assignment becomes more important.
</p>
<div class="step-flow compact-flow" aria-label="Paper roadmap">
<div><span>2</span><strong>Foundations</strong><p>Translate prompts, prefixes, tokens, and completions into states, actions, policies, and trajectories.</p></div>
<div><span>3</span><strong>Core RLHF</strong><p>Use one LLM training example to derive the reward-scoring and policy-update loop.</p></div>
<div><span>4</span><strong>Improvements</strong><p>Strengthen rewards, advantages, efficiency, and preference optimization after the basic loop is clear.</p></div>
<div><span>5-7</span><strong>Extensions</strong><p>Follow the same feedback logic into reasoning, agents, and multimodal models.</p></div>
</div>
</section>
<section class="chapter" id="preliminaries">
<p class="chapter-number">Section 2</p>
<h2>Preliminary: LLMs and RL</h2>
<p>
Before introducing policy gradients or PPO, the tutorial first makes the two sides of the story use the
same language. On the LLM side, generation is a sequence of next-token decisions conditioned on a prompt
and the prefix generated so far. On the RL side, learning is a sequence of decisions made by a policy,
evaluated by rewards along a trajectory. Section 2 connects these views so that the later algorithms can
be read as operations on familiar LLM objects.
</p>
<p>
The LLM review is intentionally selective. It covers the concepts that will later become RL variables:
token probabilities, prompts, supervised fine-tuning, decoding, and sampled outputs. It is not meant to
replace a full LLM survey; readers who want a broader background can refer to
<a href="https://arxiv.org/pdf/2501.09223" target="_blank" rel="noopener">Foundations of Large Language Models</a>.
</p>
<aside class="lesson-card">
<h3>Scope of This Section</h3>
<ul>
<li>LLM generation supplies the decision process: predict one token, append it, and continue.</li>
<li>Pre-training, prompting, supervised fine-tuning, and inference explain where the initial policy comes from.</li>
<li>MDPs, states, actions, policies, rewards, values, and trajectories explain how feedback can change that policy.</li>
<li>The bridge is the key: prompt plus prefix is the state, the next token is the action, and the complete output is a trajectory.</li>
</ul>
</aside>
<h3 id="llm-fundamentals">2.1 Fundamentals of LLMs</h3>
<p>
Modern generative LLMs are usually decoder-only Transformers. A tokenizer first converts raw text into
tokens from a vocabulary \(\mathcal{V}\), and the model assigns probability to a token sequence
\(\mathbf{z}=z_1,\ldots,z_N\) by factorizing it from left to right. This factorization matters for RL:
once a response is written as a product of next-token probabilities, changing the probability of a
sampled response can be decomposed into changing the probabilities of the tokens that formed it.
</p>
<div class="equation">
\[
\mathrm{Pr}_{\theta}(\mathbf{z})
=
\prod_{i=1}^{N}\mathrm{Pr}_{\theta}(z_i\mid\mathbf{z}_{<i}),
\qquad
\log \mathrm{Pr}_{\theta}(\mathbf{z})
=
\sum_{i=1}^{N}\log \mathrm{Pr}_{\theta}(z_i\mid\mathbf{z}_{<i})
\]
</div>
<p class="notation-note">
Here, \(\mathbf{z}_{<i}=z_1,\ldots,z_{i-1}\) denotes the tokens before \(z_i\). Causal self-attention
ensures that the prediction at each position only depends on previous tokens.
</p>
<figure class="paper-figure">
<img src="assets/figures/fig-language-modeling-and-generation.gif" alt="Diagram showing language model training and autoregressive generation">
<figcaption>Language model training and autoregressive generation.</figcaption>
</figure>
<h4>Pre-training</h4>
<p>
Pre-training teaches the model general language capabilities from large-scale unlabeled text. Each token
in a corpus naturally becomes a self-supervised target: the model observes the preceding context and
learns to predict the next token. If \(\mathcal{S}_{\mathrm{pre}}\) denotes the pre-training corpus, the
objective maximizes the total log-likelihood of all tokens.
</p>
<div class="equation">
\[
\hat{\theta}
=
\arg\max_{\theta}
\sum_{\mathbf{z}\in\mathcal{S}_{\mathrm{pre}}}
\sum_{i=1}^{N}
\log \mathrm{Pr}_{\theta}(z_i\mid\mathbf{z}_{<i})
\]
</div>
<p>
After this stage, the model can produce fluent continuations and has absorbed linguistic and factual
regularities from data. However, the objective is still to model text, not to satisfy a user's current
intention. That limitation motivates the next adaptation mechanisms: first change the input context with
prompts, then change the model parameters with supervised fine-tuning, and later change behavior with
reward-based learning.
</p>
<h4>Prompting</h4>
<p>
Prompting is a simple and lightweight way to adapt an LLM to different tasks without updating its
parameters. A prompt is the input text used to guide the model toward a desired task or output. It may
contain an instruction, user-provided content, output requirements, or demonstrations.
</p>
<aside class="callout">
<strong>Prompt example.</strong>
You are a homework assistant helping students improve their problem-solving skills. Give me three tips
to improve my accuracy in solving math problems. Please make the response clear, practical, and easy to follow.
</aside>
<p>
In this example, the prompt specifies both the task and the desired response style, and the LLM
generates the answer by continuing the input sequence. Prompts can also be written as templates. For the
same homework-assistant setting, we may keep the instruction fixed and replace only a task-specific
placeholder:
</p>
<aside class="callout">
<strong>Prompt template.</strong>
You are a homework assistant helping students improve their problem-solving skills. Give me three tips
to improve my accuracy in solving <em>{subject}</em> problems. Please make the response clear, practical,
and easy to follow.
</aside>
<p>
If <em>{subject}</em> is replaced by "math", the template becomes the prompt above. If it is replaced by
"English writing", the same instruction pattern is reused for a different task. This is why prompting is
often the first adaptation method tried before any parameter update.
</p>
<p>
Another important concept related to prompting is in-context learning. By adding demonstrations to the
context, we can let the model infer the desired input-output pattern from examples before responding to a
new input.
</p>
<aside class="callout">
<strong>In-context example.</strong><br>
<em>Input:</em> Give me two tips to improve my English writing.<br>
<em>Output:</em> 1. Read well-written essays regularly.<br>
2. Revise your sentences to make them clear.<br><br>
<em>Input:</em> Give me three tips to improve my accuracy in solving math problems.<br>
<em>Output:</em>
</aside>
<p>
When no demonstration is provided, the setting is usually called <em>zero-shot</em> prompting. When one
demonstration is included, it is called <em>one-shot</em> prompting. When several demonstrations are
included, it is called <em>few-shot</em> prompting. The key idea is that the model uses examples in the
current context to infer the expected input-output pattern without changing its parameters.
</p>
<p>
In-context learning can also be combined with chain-of-thought (CoT) prompting. CoT prompting encourages
the model to generate intermediate reasoning steps before giving the final answer. This can be done by
directly asking the model to reason step by step or by providing demonstrations that include both the
reasoning process and the answer.
</p>
<aside class="callout">
<strong>CoT example.</strong><br>
<em>Input:</em> A student solves 8 math problems and gets 6 correct. What is the accuracy? Please reason
step by step.<br>
<em>Output:</em> The student gets 6 out of 8 problems correct. The accuracy is therefore \(6/8=0.75\),
or 75%.<br><br>
<em>Input:</em> A student solves 20 math problems and gets 17 correct. What is the accuracy? Please
reason step by step.<br>
<em>Output:</em>
</aside>
<p>
Prompting adapts an LLM by changing the context used for generation. Therefore, prompt quality can
strongly affect model performance. Even for the same task, we can write the prompt in many different
ways. For example, "Give me two tips to improve my English writing." can also be written as "What are two
simple ways to improve my English writing skills?". The two prompts express nearly the same intent, but
they may lead to noticeably different outputs. Prompting is powerful because it is cheap, but it also
exposes a limit: if the model does not reliably know the desired behavior, changing the context may not
be enough. Supervised fine-tuning addresses this by updating the model on examples of desired responses.
</p>
<h4>Supervised Fine-Tuning</h4>
<p>
Supervised fine-tuning further adapts a pretrained model using labeled input-output pairs. The input
\(\mathbf{x}\) typically combines an instruction and user content, while the output \(\mathbf{y}\) is
the desired response. SFT teaches the model both what knowledge to express and how to express it in an
instruction-following format.
</p>
<table class="concept-table compact">
<thead>
<tr>
<th>Task</th>
<th>Input</th>
<th>Output</th>
</tr>
</thead>
<tbody>
<tr>
<td>Summarization</td>
<td>Summarize an article about rapid growth in solar energy.</td>
<td>Solar energy has grown rapidly and become a major renewable source.</td>
</tr>
<tr>
<td>Question answering</td>
<td>What is the capital of France?</td>
<td>Paris.</td>
</tr>
<tr>
<td>Classification</td>
<td>Classify a prize-message as spam or not spam.</td>
<td>Spam.</td>
</tr>
<tr>
<td>Machine translation</td>
<td>Translate "RL is widely used to train LLMs" into Chinese.</td>
<td>强化学习被广泛用于训练大语言模型。</td>
</tr>
</tbody>
</table>
<div class="equation">
\[
\tilde{\theta}
=
\arg\max_{\hat{\theta}^{+}}
\sum_{(\mathbf{x},\mathbf{y})\in\mathcal{S}}
\log \mathrm{Pr}_{\hat{\theta}^{+}}(\mathbf{y}\mid\mathbf{x})
\]
</div>
<p>
Here, \(\mathcal{S}\) denotes the labeled data, \(\tilde{\theta}\) denotes the parameters optimized via
SFT, and \(\hat{\theta}^{+}\) represents an adjustment to the pretrained parameters \(\hat{\theta}\).
The fine-tuning starts from the pretrained parameters rather than randomly initialized ones.
</p>
<div class="equation">
\[
\log \mathrm{Pr}_{\theta}(\mathbf{y}\mid\mathbf{x})
=
\sum_{t=1}^{T}
\log \mathrm{Pr}_{\theta}(y_t\mid\mathbf{x},\mathbf{y}_{<t})
\]
</div>
<p>
This formulation is equivalent to minimizing the cross-entropy loss over the output segment. In practice,
the input tokens are used as context and the loss is usually computed only on the output tokens. The
resulting SFT model is often the starting policy for later RL training: it can already follow the task
format, but it may still need feedback to prefer one acceptable response over another.
</p>
<h4>Inference</h4>
<p>
Inference applies the trained LLM to new inputs. Given a prompt \(\mathbf{x}\), the model predicts a
distribution over the first output token, selects a token with a decoding strategy, appends it to the
context, and repeats this autoregressive process until a stopping condition is reached.
</p>
<table class="concept-table compact">
<thead>
<tr>
<th>Decoding Strategy</th>
<th>How It Selects the Next Token</th>
<th>Tradeoff</th>
</tr>
</thead>
<tbody>
<tr>
<td>Greedy decoding</td>
<td>Selects the highest-probability token at each step.</td>
<td>Fast and deterministic, but locally optimal choices may hurt the full sequence.</td>
</tr>
<tr>
<td>Beam search</td>
<td>Keeps the top \(B\) candidate prefixes and expands them step by step.</td>
<td>Explores more candidates, but costs more compute and memory.</td>
</tr>
<tr>
<td>Sampling</td>
<td>Samples from \(\mathrm{Pr}_{\theta}(\cdot\mid\mathbf{x},\mathbf{y}_{<t})\), often with temperature, top-\(k\), or top-\(p\) filtering.</td>
<td>Produces diverse outputs and closely resembles stochastic policy sampling in RL.</td>
</tr>
</tbody>
</table>
<p>
This token-by-token generation view is the key bridge to RL. At timestep \(t\), the current context
\((\mathbf{x},\mathbf{y}_{<t})\) can be treated as the state, the selected token \(y_t\) as the action,
and the model distribution as the policy. Sampling is especially important because it produces the
candidate trajectories that later receive rewards.
</p>
<h3 id="rl-fundamentals">2.2 Fundamentals of RL</h3>
<p>
With the LLM side in place, the RL side becomes easier to read. RL provides a framework for learning
from feedback: an agent observes a state, selects an action according to a policy, receives a reward,
and transitions to a new state. For LLMs, this is most useful when exact demonstrations are hard to
specify but generated behavior can still be judged by humans, reward models, rules, verifiers, or
environments.
</p>
<figure class="paper-figure">
<img src="assets/figures/fig-llm-and-rl-elements.gif" alt="Diagram mapping LLM generation to RL elements">
<figcaption>A schematic illustration of RL for LLMs, mapping prompts, generated prefixes, tokens, rewards, and value estimates to standard RL elements.</figcaption>
</figure>
<p>
This interaction process is commonly modeled as a Markov Decision Process. A trajectory is a sequence of
states and actions, \(\tau=(s_0,a_0,s_1,a_1,\ldots,s_{H-1},a_{H-1})\), whose quality is evaluated by
accumulated rewards. The objective is to learn a policy that maximizes expected cumulative reward.
</p>
<div class="equation">
\[
\pi^{*}
=
\arg\max_{\pi}
\mathbb{E}\left[
\sum_{t=0}^{H-1}r_t
\mid \pi
\right]
\]
</div>
<p>
In LLM training, sampling means generating one or more trajectories from the current policy. A trajectory
may be a completion, a reasoning trace, a tool-use interaction, or a multimodal generation path. Once the
trajectory receives feedback, RL algorithms update the policy to make better behavior more likely.
</p>
<table class="concept-table rl-mapping-table">
<colgroup>
<col class="rl-element-col">
<col class="rl-interpretation-col">
</colgroup>
<thead>
<tr>
<th>Element</th>
<th>Interpretation</th>
</tr>
</thead>
<tbody>
<tr>
<td>Agent</td>
<td>The learner or decision-maker in RL. In the context of LLMs, the agent corresponds to the language model itself, which generates tokens sequentially and updates its behavior based on feedback signals.</td>
</tr>
<tr>
<td>Environment</td>
<td>Everything external to the agent with which it interacts. Unlike traditional RL settings that involve physical or simulated environments, the environment in LLM-based RL is typically abstract, consisting of the training framework that provides feedback, such as reward models, human annotations, or evaluation metrics, for generated outputs.</td>
</tr>
<tr>
<td>State \((s)\)</td>
<td>A state represents the current situation of the environment. For language modeling, the state at timestep \(t\) can be defined as the sequence of observed tokens up to that point, namely the context used to predict the next token. Formally, \(s_t=(\mathbf{x},\mathbf{y}_{<t})\), where \(\mathbf{x}\) denotes the input prompt and \(\mathbf{y}_{<t}\) denotes the previously generated tokens.</td>
</tr>
<tr>
<td>Action \((a)\)</td>
<td>An action corresponds to a decision made by the agent. In LLMs, actions are naturally defined as selecting the next token from the vocabulary, i.e., \(a_t=y_t\).</td>
</tr>
<tr>
<td>Reward \((r)\)</td>
<td>The reward provides feedback from the environment to evaluate the quality of an action. In general, the reward function can be defined as \(r(s,a,s')\), representing the feedback received when the agent transitions from state \(s\) to \(s'\) by taking action \(a\). At timestep \(t\), this can be written as \(r_t=r(s_t,a_t,s_{t+1})\). In deterministic settings, where the next state is uniquely determined by \((s_t,a_t)\), the reward can be simplified as \(r(s_t,a_t)\).</td>
</tr>
<tr>
<td>Policy \((\pi)\)</td>
<td>The policy defines the agent's behavior, i.e., the probability of taking an action given a state. For LLMs, the policy corresponds to the conditional probability distribution over the next token given the context: \(\pi_{\theta}(a_t\mid s_t)=\mathrm{Pr}_{\theta}(y_t\mid\mathbf{x},\mathbf{y}_{<t})\), where \(a_t=y_t\) and \(s_t=(\mathbf{x},\mathbf{y}_{<t})\). Under this formulation, an LLM can be naturally interpreted as a parameterized policy.</td>
</tr>
<tr>
<td>Value Function<br>\((V\) and \(Q)\)</td>
<td>The value function estimates the expected cumulative reward when following a policy. The state-value function \(V(s)\) measures the expected discounted return starting from state \(s\), while the action-value function \(Q(s,a)\) further conditions on the initial action.</td>
</tr>
</tbody>
</table>
<div class="equation">
\[
V(s)
=
\mathbb{E}\left[
\sum_{t=0}^{\infty}\gamma^t r_t
\mid s_0=s,\pi
\right],
\qquad
Q(s,a)
=
\mathbb{E}\left[
\sum_{t=0}^{\infty}\gamma^t r_t
\mid s_0=s,a_0=a,\pi
\right]
\]
</div>
<p class="notation-note">
These correspondences let the rest of the tutorial describe RL algorithms using familiar language-model
objects: contexts, tokens, log probabilities, rewards, values, and sampled outputs.
</p>
</section>
<section class="chapter" id="rl-training">
<p class="chapter-number">Section 3</p>
<h2>Understanding RL in LLM Training</h2>
<p>
After Section 2 maps LLM generation into RL notation, Section 3 shows how the machinery works in an
actual LLM training setting. The running example is a homework assistant whose SFT response is correct
but too short. This is a useful example because the desired improvement is easy to judge but awkward to
specify as a single gold answer. Rather than writing many new demonstrations, the tutorial defines
feedback over generated outputs and uses RL to make preferred behavior more likely.
</p>
<p>
The section develops the workflow in dependency order. Policy gradient explains how a scalar reward can
update a stochastic language model. Temporal decomposition explains how a response-level reward touches
token-level decisions. Baselines and advantages reduce noisy updates. Importance sampling and PPO make
repeated updates from sampled data more stable. Reward modeling then replaces hand-written feedback with
learned preference scores, and the pieces finally assemble into the standard PPO-based RLHF loop.
</p>
<div class="step-flow" aria-label="RLHF training loop">
<div><span>1</span><strong>Sample</strong><p>Generate outputs from the policy.</p></div>
<div><span>2</span><strong>Score</strong><p>Evaluate outputs with rewards.</p></div>
<div><span>3</span><strong>Estimate</strong><p>Compute values or advantages.</p></div>
<div><span>4</span><strong>Update</strong><p>Optimize the policy with PPO-style constraints.</p></div>
</div>
<h3 id="policy-gradient">3.1 Policy Gradient</h3>
<p>
Policy gradient methods answer the first training question: if we can only score completed samples, how
can we update the model that produced them? In LLM training, the policy is the conditional distribution
over generated tokens. Since it is impossible to enumerate all possible outputs, the expected reward is
estimated with sampled responses.
</p>
<div class="derivation">
<p class="derivation-label">From the paper's LLM notation</p>
<div class="equation">
\[
\begin{aligned}
J(\theta)
&=
\mathbb{E}_{\mathbf{x}\sim\mathcal{S}_{x}}
\left[
\sum_{\mathbf{y}\in\Omega}
\mathrm{Pr}_{\theta}(\mathbf{y}\mid\mathbf{x})
R(\mathbf{y})
\right] \\
&=
\mathbb{E}_{\mathbf{x}\sim\mathcal{S}_{x}}
\left[
\sum_{\mathbf{y}\in\Omega}
\mathrm{Pr}_{\theta}(\mathbf{y}\mid\mathbf{x})
\sum_{t=1}^{T}r_t
\right]
\end{aligned}
\]
</div>
<p>
Because the hypothesis space \(\Omega\) is too large to enumerate, the paper approximates it with
sampled outputs \(\mathcal{D}=\{\mathbf{y}_1,\ldots,\mathbf{y}_d\}\).
</p>
<div class="equation">
\[
\begin{aligned}
\frac{\partial J(\theta)}{\partial\theta}
&=
\mathbb{E}_{\mathbf{x}\sim\mathcal{S}_{x}}
\left[
\sum_{\mathbf{y}\in\mathcal{D}}
\mathrm{Pr}_{\theta}(\mathbf{y}\mid\mathbf{x})
\frac{\partial \log\mathrm{Pr}_{\theta}(\mathbf{y}\mid\mathbf{x})}{\partial\theta}
R(\mathbf{y})
\right] \\
\frac{\partial J(\theta)}{\partial\theta}
&=
\mathbb{E}_{\mathbf{x}\sim\mathcal{S}_{x}}
\left[
\frac{1}{d}
\sum_{\mathbf{y}\in\mathcal{D}}
\frac{\partial \log\mathrm{Pr}_{\theta}(\mathbf{y}\mid\mathbf{x})}{\partial\theta}
R(\mathbf{y})
\right]
\end{aligned}
\]
</div>
</div>
<div class="equation">
\[
\nabla J(\theta)
=
\mathbb{E}_{\tau\sim \pi_{\theta}}
\left[
R(\tau)\nabla_{\theta}\log \pi_{\theta}(\tau)
\right]
\]
</div>
<p class="notation-note">
Here, \(J(\theta)\) denotes the policy objective, \(\pi_{\theta}\) is the policy model parameterized by
\(\theta\), and \(\tau\) denotes a sampled trajectory. \(R(\tau)\) is the cumulative reward assigned to
the trajectory, while \(\nabla_{\theta}\log \pi_{\theta}(\tau)\) gives the gradient of the trajectory log
probability under the current policy.
</p>
<p>
The policy-gradient form is useful because the reward itself does not need to be differentiable. The
reward may be provided by a human annotator, a learned reward model, a rule-based checker, or an external
environment. The model update only requires the log probability of the sampled trajectory under the
current policy <a class="citation" href="#ref-1">[1]</a>.
</p>
<aside class="callout">
<strong>Key point.</strong>
Policy gradient does not backpropagate through the reward source. It changes the probability of sampled
behavior according to the reward assigned to that behavior.
</aside>
<figure class="paper-figure">
<img src="assets/figures/fig-policy-gradient.gif" alt="Diagram illustrating policy gradient training for LLMs">
<figcaption>Policy gradient increases the probability of sampled outputs that receive higher reward.</figcaption>
</figure>
<p>
Thus, sampled responses with higher rewards are assigned higher probability, while responses with lower
rewards are assigned lower probability. The following subsections refine this basic policy-gradient
principle to handle long sequences, noisy rewards, and unstable updates.
</p>
<h3 id="temporal-decomposition">3.2 Temporal Decomposition</h3>
<p>
The basic policy-gradient expression scores a whole response, but an LLM is updated through token
probabilities. Temporal decomposition connects these levels. Because a generated response is composed of
tokens, the probability of the full response can be decomposed into token-level probabilities, allowing
a sequence-level reward to affect the individual decisions that produced the sequence.
</p>
<div class="derivation">
<p class="derivation-label">Time-step decomposition from the paper</p>
<div class="equation">
\[
\begin{aligned}
\frac{\partial \log \mathrm{Pr}_{\theta}(\mathbf{y}\mid\mathbf{x})}{\partial\theta}
R(\mathbf{y})
&=
\left(
\sum_{t=1}^{T}
\frac{\partial \log \mathrm{Pr}_{\theta}(y_t\mid\mathbf{x},\mathbf{y}_{<t})}{\partial\theta}
\right)
\left(
\sum_{k=1}^{T}r_k
\right) \\
&=
\sum_{t=1}^{T}
\frac{\partial \log \mathrm{Pr}_{\theta}(y_t\mid\mathbf{x},\mathbf{y}_{<t})}{\partial\theta}
\left(
\sum_{k=1}^{t-1}r_k
+
\sum_{k=t}^{T}r_k
\right)
\end{aligned}
\]
</div>
<p>
At the \(t\)-th generation step, \(y_t\) does not affect the past-reward term
\(\sum_{k=1}^{t-1}r_k\). The paper therefore removes that term and keeps the future return from
\(t\) onward.
</p>
<div class="equation">
\[
\frac{\partial J(\theta)}{\partial\theta}
=
\mathbb{E}_{\mathbf{x}\sim\mathcal{S}_{x}}
\left[
\frac{1}{d}
\sum_{\mathbf{y}\in\mathcal{D}}
\sum_{t=1}^{T}
\frac{\partial \log \mathrm{Pr}_{\theta}(y_t\mid\mathbf{x},\mathbf{y}_{<t})}{\partial\theta}
\sum_{k=t}^{T}r_k
\right]
\]
</div>
</div>
<p>
The difficulty is that sequence-level rewards do not directly tell us which token caused success or
failure. A correct answer may contain many ordinary tokens that are not responsible for the reward. An
incorrect answer may be wrong because of one early reasoning error. Thus, temporal decomposition gives a
mathematical path for training, but it does not fully solve credit assignment.
</p>
<p>
This issue becomes more important as outputs become longer. In short answers, assigning the same reward
to all tokens may be acceptable. In reasoning traces and agent trajectories, a single final reward is too
coarse. Later sections therefore discuss process rewards, verifiers, and agentic credit assignment.
</p>
<h3 id="variance-reduction">3.3 Reducing Gradient Variance</h3>
<p>
Temporal decomposition gives every token a learning signal, but that signal can be very noisy. Directly
using raw rewards may increase the probability of actions simply because they appeared in a generally
good trajectory, even if those actions were not better than usual for the current state. To reduce this
variance, policy-gradient methods introduce baselines and advantages.
</p>
<div class="equation">
\[
A(s,a)=Q(s,a)-V(s)
\]
</div>
<p class="notation-note">
Here, \(A(s,a)\) denotes the advantage of taking action \(a\) at state \(s\). \(Q(s,a)\) denotes the
expected return after first taking action \(a\) at state \(s\), and \(V(s)\) denotes the expected return
from state \(s\) under the policy.
</p>
<p>
The advantage measures whether an action is better than the expected behavior at the same state. A value
model estimates the expected future reward. Subtracting this baseline keeps the expected gradient
correct while making the update more stable.
</p>
<p>
In LLMs, value models are often implemented by reusing a pretrained backbone and adding a scalar head.
They resemble reward models architecturally, but their function is different. A reward model evaluates
an output. A value model estimates the expected return of a state under the current policy.
</p>
<table class="concept-table">
<thead>
<tr>
<th>Model</th>
<th>Role in PPO-style RLHF</th>
<th>Updated?</th>
</tr>
</thead>
<tbody>
<tr><td>Policy model</td><td>Generates outputs and receives policy-gradient updates.</td><td>Yes.</td></tr>
<tr><td>Reward model</td><td>Scores generated outputs using learned preference signals.</td><td>Usually fixed during policy optimization.</td></tr>
<tr><td>Value model</td><td>Predicts expected return to estimate advantages.</td><td>Yes.</td></tr>
<tr><td>Reference model</td><td>Regularizes the policy and limits excessive drift.</td><td>No.</td></tr>
</tbody>
</table>
<h3 id="importance-sampling">3.4 Importance Sampling</h3>
<p>
Once advantages make the update less noisy, another practical issue appears: the data may already be a
little stale. Sampled outputs are often generated by an older version of the policy, while optimization
happens after one or more parameter updates. Importance sampling corrects this mismatch by using the
probability ratio between the current policy and the old policy.
</p>
<div class="equation">
\[
\rho_t(\theta)=
\frac{\pi_{\theta}(a_t\mid s_t)}
{\pi_{\theta_{\mathrm{old}}}(a_t\mid s_t)}
\]
</div>
<p class="notation-note">
Here, \(\rho_t(\theta)\) denotes the importance sampling ratio at timestep \(t\). \(\pi_{\theta}\) is
the current policy, \(\pi_{\theta_{\mathrm{old}}}\) is the old policy that generated the sampled action,
\(s_t\) is the state, and \(a_t\) is the action taken at that state.
</p>
<p>
This ratio is necessary because data freshness matters. If the learner updates too far from the policy
that produced the samples, the samples become less reliable. The paper illustrates this with a poor
sampling step that can make subsequent generations worse if the reference policy is not controlled.
</p>
<figure class="paper-figure">
<img src="assets/figures/fig-importance-terrible-step.png" alt="Diagram showing how a bad sampling step can affect future samples">
<figcaption>A bad update can affect later sampling; reference policy control limits this drift.</figcaption>
</figure>
<h3 id="ppo">3.5 Proximal Policy Optimization</h3>
<p>
Importance sampling tells the optimizer how much the current policy differs from the policy that
produced the sample. PPO adds a guardrail around that difference. Its main idea is to prevent the policy
from changing too much in a single update by clipping the importance ratio in the objective function
<a class="citation" href="#ref-4">[4]</a>.
</p>
<figure class="paper-figure">
<img src="assets/figures/fig-ppo-clip.png" alt="Diagram showing PPO clipping for positive and negative advantages">
<figcaption>PPO clips the update for both positive and negative advantages.</figcaption>
</figure>
<div class="equation">
\[
\min\left(
\rho_t A_t,
\mathrm{clip}(\rho_t,1-\epsilon,1+\epsilon)A_t
\right)
\]
</div>
<p class="notation-note">
Here, \(\rho_t\) is the probability ratio between the current and old policies, and \(A_t\) is the
advantage estimated at timestep \(t\). The clipping range \([1-\epsilon,1+\epsilon]\) limits how much
the policy can change in one update.
</p>
<p>
If the advantage is positive, the model is encouraged to increase the probability of the action, but only
within a bounded range. If the advantage is negative, the model is encouraged to reduce the probability,
again with a limit. This makes PPO suitable for RLHF, where excessive reward optimization can damage
language quality or exploit weaknesses in the reward model.
</p>
<div class="equation">
\[
\mathcal{L}(\theta)
=
-
\mathbb{E}_{\mathbf{x}\sim\mathcal{S}_{x}}
\mathbb{E}_{\mathbf{y}\sim\mathrm{Pr}_{\theta_{\mathrm{ref}}}(\cdot\mid\mathbf{x})}
\left[
\sum_{t=1}^{T}
\left(
\mathrm{Clip}\left(
\frac{\mathrm{Pr}_{\theta}(y_t\mid\mathbf{x},\mathbf{y}_{<t})}
{\mathrm{Pr}_{\theta_{\mathrm{ref}}}(y_t\mid\mathbf{x},\mathbf{y}_{<t})}
A_t
\right)
-
\beta \mathrm{Penalty}_t
\right)
\right]
\]
</div>
<p class="notation-note">
This is the paper's PPO loss: the clipped advantage term updates the policy, while
\(\beta \mathrm{Penalty}_t\) constrains drift from the reference model.
</p>
<aside class="callout">
<strong>Penalty term.</strong>
In the paper, the time-step penalty is
\[
\mathrm{Penalty}_t
=
\log \mathrm{Pr}_{\theta}(y_t\mid\mathbf{x},\mathbf{y}_{<t})
-
\log \mathrm{Pr}_{\theta_{\mathrm{ref}}}(y_t\mid\mathbf{x},\mathbf{y}_{<t}).
\]
It measures how far the current policy moves from the reference policy on the same sampled token.
The coefficient \(\beta\) is the weight of this penalty.
</aside>
<h3 id="reward-models">3.6 Training Reward Models</h3>
<p>
Up to this point, the reward can be imagined as a simple score. RLHF needs a scalable way to produce
that score for many sampled responses, so it usually trains a reward model before policy optimization.
A reward model maps an input-output pair to a scalar score and is commonly trained from pairwise
preference data, where a human or AI judge chooses the better response among two candidates
<a class="citation" href="#ref-5">[5]</a>.
</p>
<p>
The paper first describes several ways to collect feedback. Rating asks annotators to assign numerical
scores. Listwise ranking asks them to order a set of candidate outputs. Pairwise comparison asks them to
choose the better of two outputs for the same input. Pairwise comparison is widely used because it is
usually easier and more reliable than assigning absolute scores.
</p>
<div class="equation">
\[
\mathcal{L}_{\mathrm{RM}}
=
-\log \sigma\left(r_{\phi}(\mathbf{x},\mathbf{y}_w)-r_{\phi}(\mathbf{x},\mathbf{y}_l)\right)
\]
</div>
<p class="notation-note">
Here, \(\mathcal{L}_{\mathrm{RM}}\) denotes the reward model loss, \(r_{\phi}\) is the reward model
parameterized by \(\phi\), \(\mathbf{x}\) is the input, \(\mathbf{y}_w\) is the preferred output
sequence, and \(\mathbf{y}_l\) is the rejected output sequence. The sigmoid function turns the reward
difference into a preference probability.
</p>
<p>
Pairwise comparison is commonly used because it is often more reliable to annotate than absolute
numerical scores. Once trained, the reward model can score many newly generated responses, providing a
scalable signal for policy optimization.
</p>
<figure class="paper-figure">
<img src="assets/figures/fig-rm-pairwise.gif" alt="Diagram showing pairwise reward model loss with preferred and rejected outputs">
<figcaption>Reward models learn to assign higher scores to preferred responses.</figcaption>
</figure>
<p>
During policy optimization, the reward model usually scores a complete input-output pair independently.
This means the reward is often sparse: intermediate tokens receive no direct reward, and the final token
receives the reward model score. This limitation is one reason later sections discuss reward shaping,
process reward models, and step-level verification.
</p>
<h3 id="complete-workflow">3.7 A Complete Workflow</h3>
<p>
The section concludes by combining the components into a standard PPO-based RLHF workflow. First, a
reward model is trained from preference data. Then the policy samples outputs. The reward model scores
these outputs, a value model estimates advantages, and PPO updates the policy under a reference policy
constraint.
</p>
<figure class="paper-figure">
<img src="assets/figures/fig-ppo-workflow.gif" alt="Diagram showing PPO-based optimization workflow for RLHF">
<figcaption>The PPO-based RLHF loop combines sampling, reward scoring, value estimation, and policy update.</figcaption>
</figure>
<p>
This workflow explains why RLHF is not only an algorithmic objective. It is a training system involving
data generation, reward inference, KL regularization, advantage estimation, optimization, and evaluation.
Once the system view is clear, the next chapter becomes natural: many modern methods can be understood
as attempts to make one component of this loop more reliable, cheaper, or easier to optimize
<a class="citation" href="#ref-2">[2]</a>.
</p>
<aside class="recap-card">
<h3>Section Recap</h3>
<p>
PPO-style RLHF is best understood as a system: the policy samples, the reward model scores, the value
model estimates advantages, and the reference model keeps optimization from drifting too far.
</p>
</aside>
</section>
<section class="chapter" id="improved-rl">
<p class="chapter-number">Section 4</p>
<h2>Improved RL for LLMs</h2>
<p>
Section 3 builds the basic PPO-style RLHF workflow. Section 4 keeps the same workflow in view and asks
what must be improved before it becomes practical at modern LLM scale. The discussion is organized
around four pressure points: how to construct better rewards, how to estimate advantages without
excessive variance or cost, how to spend less computation during online training, and when to replace
the explicit reward-model-plus-RL pipeline with direct preference learning.
</p>
<table class="concept-table">
<thead>
<tr>
<th>Problem</th>
<th>Methods Discussed</th>
<th>Main Benefit</th>
</tr>
</thead>
<tbody>
<tr><td>Sparse or weak rewards</td><td>Reward shaping, rule-based rewards, process rewards.</td><td>Provides denser or more reliable feedback.</td></tr>
<tr><td>Reward model limitations</td><td>Generative reward models, rubric-based rewards, reward evaluation.</td><td>Improves alignment and evaluation transparency.</td></tr>
<tr><td>Expensive or unstable critics</td><td>TD, GAE, GRPO, value-free advantage estimation.</td><td>Improves the bias-variance and system-cost tradeoff.</td></tr>
<tr><td>Pipeline complexity</td><td>DPO and related direct preference methods.</td><td>Optimizes from preference pairs without an explicit online RL loop.</td></tr>
</tbody>
</table>
<p>
The order of the chapter mirrors the training loop. It starts with the reward because the reward defines
what the policy will learn. It then moves to advantage estimation because rewards must be converted into
stable token-level updates. Efficiency comes next because sampling and scoring dominate practical cost.
DPO is discussed last because it changes the pipeline itself.
</p>
<h3 id="advanced-reward-models">4.1 Advanced Reward Models</h3>
<p>
Reward construction strongly determines what the policy learns. If the reward is sparse, noisy, biased,
or vulnerable to exploitation, policy optimization may produce behavior that scores well without matching
the true objective. Therefore, modern LLM RL places increasing emphasis on reward design.
</p>
<h3 id="automatic-preference-data">4.1.1 Automatic Preference Data Generation</h3>
<p>
The first issue is data. Human preference labels are useful, but they are expensive and subjective. A
scalable alternative is to ask a strong model or verifier to compare candidate outputs. In the
homework-assistant example from the paper, the judge sees the input and two candidate answers, then
selects the one that is more informative, accurate, clear, and helpful.
</p>
<p>
This makes preference data easier to scale because the system can generate both outputs and labels.
However, the paper also stresses the risk: AI feedback can inherit position bias, judge-specific
preferences, and mistakes in the judging prompt. In practice, prompt design, demonstrations, diverse
candidate generation, and mixing human and AI feedback are used to improve the quality and diversity of
generated preference data.
</p>