Two stages, in one order:
gemini-3.5-flash
│
│ STAGE 1 — supervised fine-tuning
│ 3,631 (prompt, poem) pairs from the Diwan
│ teaches the DISTRIBUTION: what folk poetry sounds like
▼
tunifolk-sft-base
│
│ STAGE 2 — reinforcement fine-tuning
│ 843 prompts, no answers, graded by a reward
│ teaches the CONSTRAINT: rhyme topology, dialect, novelty
▼
tunifolk-rlft-from-sft
The two objectives can express different things, and neither substitutes for the other.
Supervised tuning can only teach what a target exemplifies. Every example is a poem the tradition actually produced, and the loss says "produce this". That is exactly right for moving register — diction, imagery, the sound of the Mudawwana — because those properties are demonstrated by the corpus. It is the wrong instrument for "satisfy this rhyme topology and stay in dialect and do not copy the source", because no single poem demonstrates the conjunction, and copying a training poem is zero loss.
Reinforcement tuning can only teach what a scorer can grade. It never sees a target, so it can be pushed toward a conjunction of properties no exemplar holds, and it can be told that copying is worthless. But starting it from the base model means spending the early run teaching register that a supervised pass learns in an hour.
So: SFT for what the corpus shows, RL for what a program can check.
The service enforces it. Submitting a supervised job whose preTunedModel is a
reinforcement-tuned model is refused:
400 INVALID_ARGUMENT
The pre-tuned model … has an unsupported tune type Reinforcement Tuning.
Continuous tuning of type Supervised Fine-Tuning only supports the following
pre-tuned model tune types: Supervised Fine-Tuning
The documentation does not state this. Its continuous-tuning pages list
supervised → reinforcement and reinforcement → reinforcement as supported,
and are silent on the reverse. The constraint was established by submitting the
request and reading the error.
Reinforcement tuning is terminal. Nothing can be chained after it. If both stages are wanted, SFT must come first — which happens to be the order the argument above already recommends.
Measured against the ablation: the same reinforcement stage, same dataset, same
hyperparameters, started from base gemini-3.5-flash instead
(configs/baseline_rl_from_base.yaml, job tunifolk-rlft-v5, ran to
completion).
| baseline: base → RL | pipeline: base → SFT → RL | |
|---|---|---|
structure at step 0 |
0.4229 | 0.7624 |
/train_mean_reward final |
0.9154 (step 158) | 0.9250 (step 86, still running) |
structure final |
0.8805 (step 158) | 0.9246 (step 86) |
/eval_mean_reward |
0.9025 (step 159) | 0.9216 (step 78) |
| generation length | 12,782 tokens | 927 tokens |
| wall clock | 2 d 2 h | ~1 h (SFT) + in progress |
Three things to read from that.
The supervised stage transfers. Reinforcement starts at structure 0.762
instead of 0.423 — the RL stage begins where the baseline was roughly a third of
the way through.
It does not merely start higher, it ends higher. At step 86 of 158 the
pipeline has already passed the baseline's final score on every headline
series. The gap is largest on structure (0.925 vs 0.881), which is the
deterministic component, not the judge.
Output is 13.8× shorter. 927 tokens against 12,782. That is the thinking
budget: the baseline runs thinkingLevel: HIGH and spends ~98% of its output on
thinking tokens, while the pipeline's RL stage runs MINIMAL and spends
essentially all of it on the poem. Same quality of result, an order of magnitude
less generation — which is cost, latency, and the reason the run finishes sooner.
Neither is optional, and both were found the hard way.
thinkingLevel: HIGH is broken on a supervised-tuned policy. Measured on the
stage-1 endpoint, same prompt, maxOutputTokens: 32768:
| thinkingLevel | latency | output tokens | thinking tokens | result |
|---|---|---|---|---|
MINIMAL |
8.0 s | 481 | 0 | a poem, 1,122 chars |
LOW |
7.7 s | 784 | 0 | a poem, 1,808 chars |
HIGH |
129.0 s | 1 | 31,457 | 1 character |
HIGH (repeat) |
16.1 s | 0 | 1,643 | 0 characters |
Supervised tuning teaches the model to emit the target directly, omitting the thinking pass — which is why the SFT documentation recommends minimising thinking on tuned tasks. Forced to think anyway, this policy exhausts its budget and returns nothing. Under reinforcement tuning that is a floor reward of −1.0 on nearly every sample: a multi-day run learning from noise.
Stage 2 therefore sets thinkingLevel: MINIMAL. The baseline ablation keeps
HIGH, which is correct for it — a base model does benefit from deliberating
its way to a rhyme topology.
tunisianity is a contrastive component: it scores "more corpus-like than
this model's output", learned from negatives generated by the policy the
reward will grade. Stage 1 deliberately moves that policy toward the corpus,
which is precisely the distribution the discriminator was built to separate
from it.
Refitting against stage-1 output, held-out AUC falls from 0.9999 to 0.945, and the corpus/policy median gap narrows from wide to 0.936 vs 0.845. Still far above the 0.60 gate — but the old discriminator would have been scoring a distribution that no longer exists.
A second effect is more interesting. form AUC flips from 0.320 to 0.804.
Against the base model, form scored below chance because an LLM asked for
aaaB quatrains out-regularises an oral tradition. The supervised model does
not: it sits at 0.741 against a corpus median of 1.000. So in the pipeline the
saturating form constraint has real headroom and supplies genuine gradient,
where against the base model it was already saturated at step 0.
Stage 2 is therefore always preceded by make rl-negatives (against the stage-1
endpoint), make rl-tunisianity, make rl-acceptance, and a redeploy of the
reward service. make stage2 chains all of it.
| stage 1 (SFT) | stage 2 (RL) | |
|---|---|---|
| Data | 3,631 (prompt, poem) pairs | 843 prompts, no answers |
| Extra infrastructure | none | Cloud Run reward + LLM judge |
| Hyperparameters | 3 | 8 |
| API surface | v1, supervisedTuningSpec |
v1beta1, reinforcementTuningSpec |
| Wall clock | ~1 hour | ~1–2 days |
| Guards against | — | reward hacking |
| Vulnerable to | memorisation | reward hacking |
Their weaknesses are complementary, which is the second argument for running
both: supervised tuning's characteristic failure is reproducing a training poem
verbatim, and stage 2's novelty component penalises exactly that.