Skip to content

Latest commit

 

History

History
160 lines (126 loc) · 6.94 KB

File metadata and controls

160 lines (126 loc) · 6.94 KB

03 — The pipeline

Two stages, in one order:

gemini-3.5-flash
      │
      │  STAGE 1 — supervised fine-tuning
      │  3,631 (prompt, poem) pairs from the Diwan
      │  teaches the DISTRIBUTION: what folk poetry sounds like
      ▼
tunifolk-sft-base
      │
      │  STAGE 2 — reinforcement fine-tuning
      │  843 prompts, no answers, graded by a reward
      │  teaches the CONSTRAINT: rhyme topology, dialect, novelty
      ▼
tunifolk-rlft-from-sft

Why two stages rather than one

The two objectives can express different things, and neither substitutes for the other.

Supervised tuning can only teach what a target exemplifies. Every example is a poem the tradition actually produced, and the loss says "produce this". That is exactly right for moving register — diction, imagery, the sound of the Mudawwana — because those properties are demonstrated by the corpus. It is the wrong instrument for "satisfy this rhyme topology and stay in dialect and do not copy the source", because no single poem demonstrates the conjunction, and copying a training poem is zero loss.

Reinforcement tuning can only teach what a scorer can grade. It never sees a target, so it can be pushed toward a conjunction of properties no exemplar holds, and it can be told that copying is worthless. But starting it from the base model means spending the early run teaching register that a supervised pass learns in an hour.

So: SFT for what the corpus shows, RL for what a program can check.

Why this order is forced, not chosen

The service enforces it. Submitting a supervised job whose preTunedModel is a reinforcement-tuned model is refused:

400 INVALID_ARGUMENT
The pre-tuned model … has an unsupported tune type Reinforcement Tuning.
Continuous tuning of type Supervised Fine-Tuning only supports the following
pre-tuned model tune types: Supervised Fine-Tuning

The documentation does not state this. Its continuous-tuning pages list supervised → reinforcement and reinforcement → reinforcement as supported, and are silent on the reverse. The constraint was established by submitting the request and reading the error.

Reinforcement tuning is terminal. Nothing can be chained after it. If both stages are wanted, SFT must come first — which happens to be the order the argument above already recommends.

Does it work?

Measured against the ablation: the same reinforcement stage, same dataset, same hyperparameters, started from base gemini-3.5-flash instead (configs/baseline_rl_from_base.yaml, job tunifolk-rlft-v5, ran to completion).

baseline: base → RL pipeline: base → SFT → RL
structure at step 0 0.4229 0.7624
/train_mean_reward final 0.9154 (step 158) 0.9250 (step 86, still running)
structure final 0.8805 (step 158) 0.9246 (step 86)
/eval_mean_reward 0.9025 (step 159) 0.9216 (step 78)
generation length 12,782 tokens 927 tokens
wall clock 2 d 2 h ~1 h (SFT) + in progress

Three things to read from that.

The supervised stage transfers. Reinforcement starts at structure 0.762 instead of 0.423 — the RL stage begins where the baseline was roughly a third of the way through.

It does not merely start higher, it ends higher. At step 86 of 158 the pipeline has already passed the baseline's final score on every headline series. The gap is largest on structure (0.925 vs 0.881), which is the deterministic component, not the judge.

Output is 13.8× shorter. 927 tokens against 12,782. That is the thinking budget: the baseline runs thinkingLevel: HIGH and spends ~98% of its output on thinking tokens, while the pipeline's RL stage runs MINIMAL and spends essentially all of it on the poem. Same quality of result, an order of magnitude less generation — which is cost, latency, and the reason the run finishes sooner.

Two constraints the pipeline imposes on stage 2

Neither is optional, and both were found the hard way.

Thinking must be turned down

thinkingLevel: HIGH is broken on a supervised-tuned policy. Measured on the stage-1 endpoint, same prompt, maxOutputTokens: 32768:

thinkingLevel latency output tokens thinking tokens result
MINIMAL 8.0 s 481 0 a poem, 1,122 chars
LOW 7.7 s 784 0 a poem, 1,808 chars
HIGH 129.0 s 1 31,457 1 character
HIGH (repeat) 16.1 s 0 1,643 0 characters

Supervised tuning teaches the model to emit the target directly, omitting the thinking pass — which is why the SFT documentation recommends minimising thinking on tuned tasks. Forced to think anyway, this policy exhausts its budget and returns nothing. Under reinforcement tuning that is a floor reward of −1.0 on nearly every sample: a multi-day run learning from noise.

Stage 2 therefore sets thinkingLevel: MINIMAL. The baseline ablation keeps HIGH, which is correct for it — a base model does benefit from deliberating its way to a rhyme topology.

The discriminator must be refit

tunisianity is a contrastive component: it scores "more corpus-like than this model's output", learned from negatives generated by the policy the reward will grade. Stage 1 deliberately moves that policy toward the corpus, which is precisely the distribution the discriminator was built to separate from it.

Refitting against stage-1 output, held-out AUC falls from 0.9999 to 0.945, and the corpus/policy median gap narrows from wide to 0.936 vs 0.845. Still far above the 0.60 gate — but the old discriminator would have been scoring a distribution that no longer exists.

A second effect is more interesting. form AUC flips from 0.320 to 0.804. Against the base model, form scored below chance because an LLM asked for aaaB quatrains out-regularises an oral tradition. The supervised model does not: it sits at 0.741 against a corpus median of 1.000. So in the pipeline the saturating form constraint has real headroom and supplies genuine gradient, where against the base model it was already saturated at step 0.

Stage 2 is therefore always preceded by make rl-negatives (against the stage-1 endpoint), make rl-tunisianity, make rl-acceptance, and a redeploy of the reward service. make stage2 chains all of it.

What each stage costs

stage 1 (SFT) stage 2 (RL)
Data 3,631 (prompt, poem) pairs 843 prompts, no answers
Extra infrastructure none Cloud Run reward + LLM judge
Hyperparameters 3 8
API surface v1, supervisedTuningSpec v1beta1, reinforcementTuningSpec
Wall clock ~1 hour ~1–2 days
Guards against reward hacking
Vulnerable to memorisation reward hacking

Their weaknesses are complementary, which is the second argument for running both: supervised tuning's characteristic failure is reproducing a training poem verbatim, and stage 2's novelty component penalises exactly that.