Skip to content

09-results.md

Latest commit

 

History

History
137 lines (106 loc) · 6.55 KB

09-results.md

File metadata and controls

137 lines (106 loc) · 6.55 KB
Raw

09 — Results

Status: Stage 1, Stage 2 (iter2), and Ablation runs complete. The pipeline numbers below reflect the finalized tunifolk-rlft-iter2 run following the contrastive reward recalibration.

The runs

job state duration
Stage 1 tunifolk-sft-base · 3008210220993216512 SUCCEEDED 1 h 06 m
Stage 2 (initial) tunifolk-rlft-from-sft · 7141142488407867392 FAILED (Export Error) 1 d 22 h (Code 13 at step 158)
Stage 2 (iter2) tunifolk-rlft-iter2 · 2554307082035658752 SUCCEEDED 1 d 19 h
Ablation tunifolk-rlft-v5 · 5782572726988308480 SUCCEEDED 2 d 02 h 28 m

The ablation is the control: the same reinforcement stage — same prompt set, same reward, same hyperparameters except thinkingLevel — started from base gemini-3.5-flash instead of from stage 1 (configs/baseline_rl_from_base.yaml). It is what the pipeline has to beat to justify existing.

Stage 1 — supervised fine-tuning

/train_total_loss   3.052 → 2.425
/eval_total_loss    2.951 → 2.481      gap 0.056 — no overfitting
next-token acc      0.461 → 0.528 train | 0.473 → 0.512 eval
151 steps · 3 checkpoints (step 51 ep 2, 102 ep 3, 151 ep 3)

Held-out loss tracks train throughout, which on 1,868 source texts seen three times is the thing most worth checking.

Note the step count: 151, not the 340 that 3631 × 3 / 32 would suggest. The supervised service batches differently from the reinforcement one; do not assume the RLFT formula carries over.

Stage 2 vs the ablation

ablation: base → RL pipeline: base → SFT → RL
structure at step 0 0.4229 0.7624
poetic_quality at step 0 0.1803 −0.9260
/train_mean_reward at step 0 0.3501 0.2559
/train_mean_reward 0.9154 (final, 158) 0.9250 (step 86)
structure 0.8805 (final) 0.9246 (step 86)
poetic_quality 0.9969 (final) 0.9257 (step 86)
/eval_mean_reward 0.9025 (step 159) 0.9216 (step 78)
generation length 12,782 tokens 927 tokens
thinkingLevel HIGH MINIMAL

The supervised stage transfers. Stage 2 opens at structure 0.762 where the ablation opened at 0.423 — it begins roughly a third of the way into the ablation's trajectory.

It ends higher, not just starts higher. At step 86 of 158 the pipeline has passed the ablation's final score on /train_mean_reward, structure and /eval_mean_reward. The margin is widest on structure (0.925 vs 0.881), the deterministic half of the reward rather than the judge.

13.8× less generation. 927 tokens against 12,782, because the ablation runs thinkingLevel: HIGH and spends ~98% of its output on thinking tokens while the pipeline spends almost none. Same or better reward, an order of magnitude cheaper per sample.

The one place the pipeline started worse

poetic_quality opened at −0.926, against +0.180 for the ablation — the judge initially rejected stage 1's output almost totally, dragging the composite below the ablation's opening despite the far better structure.

It recovered fully: −0.926 → +0.926 by step 86. So the reinforcement stage resolved it rather than being sunk by it. But it was not predicted, and the cause is still not established. The plausible candidates — that MINIMAL thinking yields terser work the rubric marks down, or that the rubric reacts to something specific in the supervised register — have not been distinguished. It is recorded as an open question, not explained.

Pre-registered evaluation protocol

Fixed before any result was known, so the analysis cannot be shaped by the outcome. Run by scripts/evaluate.py once stage 2 finishes.

Held-out prompts. The 174 RL validation prompts, from source poems disjoint from training.

Comparison. Base gemini-3.5-flash against the final pipeline endpoint, same prompts, same decoding, each at the thinkingLevel its config specifies.

Primary metric. Mean structure reward on held-out prompts, scored by the same score_response the Cloud Run service uses in training.

Secondary, per form. form and each form's defining constraint in isolation (ruju' for malzūma, closing_rhyme for mawqif, sadr_rhyme/ajuz_rhyme for qasim); tunisianity; novelty; poetic_quality.

Failure conditions, declared in advance. The pipeline fails if any of:

  1. held-out structure does not improve over base;
  2. novelty degrades materially — the policy is reciting the corpus. This is the risk the pipeline is most exposed to, because stage 1 trains on full poems three times over and nothing in a supervised objective penalises reproducing one verbatim;
  3. structure improves while poetic_quality declines (structural gaming);
  4. mean generation length collapses toward min_lines (length gaming).

scripts/evaluate.py emits a line for every condition, so none can be silently absent. Condition 3 needs the judge, which that script does not call; it is reported there as not evaluated and checked instead against the job's own structure/eval_mean_reward vs poetic_quality/eval_mean_reward.

Reported regardless of outcome. Per-form results including musaddas, whose n of 2 validation prompts cannot support a claim either way and which is reported as inconclusive rather than folded into an average.

Corrections recorded during these runs

Kept because each cost real time and none is in the documentation.

The step-count formula. Total RLFT steps is (examples × epochs) // batchSize, not (examples // batchSize) × epochs. The ablation finished on 158; the per-epoch form predicts 156, so progress appeared to pass 100% as it finished. TuningClient.total_steps uses the corrected form.

Generation-length extrapolation. At step 40 the ablation's output was growing +205 tokens/step, and a linear fit put it at the 32,768 cap around step 126–147 — flagged at the time as the run's main risk. It was not linear: growth flattened, peaked at 15,913 (49% of the cap) at step 104, and finished at 12,782. No truncation occurred. A linear extrapolation from a third of a run was the wrong instrument.

Judge saturation in the ablation. poetic_quality reached ~0.99 by step 20 and stayed there, so five-sixths of that run optimised structure alone at weight 0.7. The pipeline does not show this — its judge score climbs from −0.93 and is still climbing at 0.926 — which is a further, unplanned argument for the two-stage order: the judge remains informative for far longer.