An end-to-end SFT → RL pipeline that teaches Gemini 3.5 Flash to compose Tunisian folk poetry (الشعر الشعبي التونسي / الملحون).
gemini-3.5-flash
│
│ STAGE 1 — supervised fine-tuning
│ 3,631 (prompt, poem) pairs from the Diwan
│ teaches the DISTRIBUTION — what folk poetry sounds like
▼
tunifolk-sft-base
│
│ STAGE 2 — reinforcement fine-tuning
│ 843 prompts, no answers, graded by a programmatic reward
│ teaches the CONSTRAINT — rhyme topology, dialect, novelty
▼
tunifolk-rlft-from-sft
Tunisian folk poetry is an unusually good tuning target. Its metrical system is
not Khalilian — Muhammad al-Marzuqi's argument, adopted by Muhyi al-Din
Khrayyef and by the editorial committee of the national corpus, is that the
Tunisian dialect has asbab and awtad but no fawasil, so classical buhur
cannot be applied at all:
«الشعر الشعبي الحديث لا يمكن أن نطبّق عليه البحور القديمة… ولا يمكن أن نضبط موازينه إلا بواسطة الإيقاع» — المرزوقي، الأدب الشعبي، ص 83
What does hold a poem together is a rhyme topology that differs per form and is fully decidable by a program. That is what makes a reinforcement reward possible here at all: an objective, cheap, hard-to-game structural signal on a creative task.
The two stages teach different things, and neither substitutes for the other:
- Supervised tuning can only teach what a target exemplifies. Right for register — diction, imagery, the sound of the Mudawwana. Wrong for "satisfy this topology and stay in dialect and don't copy the source", because no single poem demonstrates the conjunction, and copying a training poem is zero loss.
- Reinforcement tuning can only teach what a scorer can grade. It never sees a target, so it can be pushed toward that conjunction and told copying is worthless — but from the base model it wastes its early run learning register that a supervised pass learns in an hour.
The order is enforced, not chosen. A supervised job cannot start from a reinforcement-tuned model — the service refuses it. Reinforcement tuning is terminal. See 03 — The pipeline.
Against the ablation — identical reinforcement stage, same data and
hyperparameters, started from base gemini-3.5-flash instead:
| baseline: base → RL | pipeline: base → SFT → RL | |
|---|---|---|
structure at step 0 |
0.4229 | 0.7624 |
/train_mean_reward |
0.9154 (final, step 158) | 0.9250 (step 86, still running) |
structure |
0.8805 (final) | 0.9246 (step 86) |
/eval_mean_reward |
0.9025 | 0.9216 |
| generation length | 12,782 tokens | 927 tokens |
At barely half-trained the pipeline has already passed the baseline's final score on every headline series, and produces its answers in 13.8× fewer tokens — the baseline spends ~98% of its output on thinking, the pipeline almost none.
Every structural reward component derives from the rhyme topology of the four
usul:
| Form | Topology | Defining constraint |
|---|---|---|
القسيم qasim |
a B / a B / a B … |
two parallel monorhymes, no tali', no return |
الملزومة malzuma |
AA ‖ bbb A / ccc A … |
every strophe returns (ruju') to the tali' rhyme — this is what malzūma means |
الموقف mawqif |
aaaB / cccB … |
fourth ghusn holds one rhyme fixed across the whole poem |
المسدّسة musaddas |
AAA ‖ bbbb AA … |
sextets whose last two aghsan return to the tali' |
Implemented in src/tunifolk/prosody/forms.py.
The most important thing to understand about the stage-2 reward. Components have three different jobs and cannot share one acceptance criterion:
| Component | AUC vs base | AUC vs SFT | Role | How it is verified |
|---|---|---|---|---|
tunisianity |
1.000 | 0.998 | discriminator | AUC ≥ 0.60 |
form |
0.320 | 0.804 | constraint | scrambling a real poem must lower it (0.923 → 0.513) |
novelty |
0.500 | 0.500 | guard | submitting the source verbatim must collapse it (→ 0.000) |
non_repetition |
0.397 | 0.822 | guard | a repeated line must collapse it (→ 0.086) |
meter |
0.268 | 0.522 | diagnostic | weight 0; reported, not optimised |
Against the base model, form scores below chance — an LLM asked for aaaB
quatrains out-regularises an oral tradition, so a monotonic structural reward
would push the policy away from the target register. form and meter
therefore saturate at per-form corpus medians: full credit for reaching the
tradition's level, nothing for exceeding it.
Note the form column flips to 0.804 after stage 1. The supervised model does
not out-regularise the corpus, so in the pipeline that constraint has real
headroom and supplies genuine gradient. Full reasoning in
07 — Reward.
src/tunifolk/
├── prosody/ shared measurement core, used by both stages
│ ├── normalize.py orthographic normalisation (never "corrects" dialect to MSA)
│ ├── rhyme.py rawiyy extraction + rhyme agreement
│ ├── forms.py the four usul as structural validators
│ ├── meter.py positional length regularity (a prior, not a scansion)
│ ├── tunisianity.py contrastive char-4gram LLR vs the current policy
│ └── novelty.py anti-plagiarism fingerprints
├── data/
│ ├── diwan.py markdown Diwan, full text -> stage 1
│ └── corpus.py classified JSON export -> stage 2
├── sft/dataset.py stage 1: (prompt, poem) pairs
├── rl/ stage 2: reward, Cloud Run service, judge, prompt set
└── tuning/client.py one job client: v1 + v1beta1, metrics for both stages
artifacts/ and docs/ follow the same two-stage shape.
make venv # dev environment
make test # 105 unit tests
make help # every target, grouped by stage
make pipeline # stage 1 then stage 2, end to endOr stage by stage:
# Stage 1 — supervised
make sft-dataset DIWAN=/path/to/diwan # 3,631 (prompt, poem) pairs
make sft-upload && make sft-launch
make sft-status
# Stage 2 — reinforcement (recalibrates against stage 1 first)
make rl-negatives CORPUS=/path/classified.json # adversary from stage 1's endpoint
make rl-tunisianity # refit the discriminator
make rl-acceptance # GATE
make rl-dataset && make rl-upload
make rl-deploy # reward scorer
make rl-launch && make rl-status
make baseline # the ablation: RL straight from base, skipping stage 1Stage 2 resolves stage 1's endpoint automatically — rl-negatives generates the
adversary from the policy the reward will actually grade. Reusing base-model
negatives would calibrate the discriminator against a distribution that no
longer exists; see 03 — The pipeline.
- A GCP project with
aiplatformenabled; plusrun,cloudbuildandartifactregistryfor the stage-2 reward service. - Stage 2 only:
roles/run.invokerforservice-<PROJECT_NUMBER>@gcp-sa-vertex-tune.iam.gserviceaccount.comon the reward service — the tuning agent is the caller. - The corpus. Not vendored — see 02 — The corpus.
docs/ — read 01 — Background then 03 — The pipeline.
| 01 — Background | the tradition, the forms, the sources |
| 02 — The corpus | the data, its two views, what ships |
| 03 — The pipeline | why SFT then RL, why the order is forced, and what it buys |
| 04 — SFT dataset | stage 1: (prompt, poem) pairs |
| 05 — SFT running | stage 1: three hyperparameters, thinking turned down |
| 06 — RL dataset | stage 2: prompt-only examples |
| 07 — RL reward | stage 2: components, roles, blind spots |
| 08 — RL running | stage 2: config, monitoring, API edges |
| 09 — Results | pipeline vs ablation, pre-registered protocol |
Both datasets are committed, so either stage reproduces from this repository alone:
artifacts/sft/dataset/ 3,631 train + 311 validation (prompt, poem) pairs
artifacts/rl/dataset/ 843 train + 174 validation prompts, no answers
artifacts/rl/reward/ the adversary and fitted discriminator behind the gate
No corpus text is committed. Both corpus views are supplied by path at runtime. Note the asymmetry between stages: an RL example exposes at most one opening hemistich and a hash fingerprint, while an SFT example necessarily contains a full poem, because that is its target.
The one derived model that ships is a 350 KB contrastive n-gram file. Chaining its 4-grams back into running text yields gibberish, so poems are not extractable from it; 20 of a given line's 26 4-grams are present, making it a membership oracle rather than an extraction oracle.
Apache-2.0 for the code in this repository.