Reinforcement-learning fine-tuning of Gemini 3.5 Flash for Tunisian folk poetry

Tunisian folk poetry is a rare RLFT target: its metrical system is not
Khalilian, so what organises a poem is a rhyme topology that differs per form
and is fully decidable by a program. That yields an objective, cheap,
hard-to-game structural reward on a task where supervised fine-tuning could
only teach imitation of 843 specific poems.

The reward is a composite of 0.7 x a deterministic structural scorer served
from Cloud Run and 0.3 x a Gemini judge, kept as separate named rewards so the
two curves can be watched diverging.

Its central design point is that components have three different jobs and
cannot share one acceptance criterion:

  - tunisianity discriminates (contrastive char-4gram LLR, held-out AUC 0.9999
    vs the base model, 0.9875 vs MSA on an axis it was never trained on);
  - form and meter are saturating constraints whose AUC is below chance by
    construction, because the base model out-regularises the oral tradition;
    they are gated on a form-violation check, not on AUC;
  - novelty and non-repetition are guards, verified by constructing the attack
    they exist to stop rather than by any AUC.

Every component is calibrated against the adversary that matters -- 520
base-model attempts at the real prompts -- never against line-shuffled or
cross-form text, which any metric separates trivially.

Includes the RLFT datasets. artifacts/dataset/{train,validation}.jsonl are the
exact bytes uploaded to GCS and consumed by the tuning job, so a run is
reproducible from this repository alone. No corpus text is redistributed: a
prompt carries at most one opening hemistich and a hash fingerprint.

The tuning client reads training metrics over REST through the experiment's
backing Tensorboard, which is the only way to obtain a live training step --
the tuningJobs resource exposes no step or progress field.

Author: cnemri

.dockerignore

@@ -0,0 +1,7 @@
+.venv/
+artifacts/
+tests/
+docs/
+scripts/
+.git/
+__pycache__/

.github/workflows/ci.yaml

@@ -0,0 +1,14 @@
+name: ci
+on:
+  push: { branches: [main] }
+  pull_request:
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with: { python-version: "3.12" }
+      - run: pip install -e ".[dev]"
+      - run: ruff check src tests scripts
+      - run: pytest -q

.gitignore

@@ -0,0 +1,16 @@
+.venv/
+__pycache__/
+*.py[cod]
+*.egg-info/
+.pytest_cache/
+.ruff_cache/
+.env
+
+# macOS junk
+.DS_Store
+**/.DS_Store
+
+# Everything under artifacts/ is tracked. It is the experimental record, and it
+# includes the RLFT datasets: artifacts/dataset/train.jsonl and validation.jsonl
+# are the exact bytes uploaded to GCS and consumed by the tuning job, so a run
+# is reproducible from the repository alone. Total is a few megabytes.

Dockerfile

@@ -0,0 +1,18 @@
+# Reward scorer for RLFT. Runs the same installed package the tests exercise.
+FROM python:3.12-slim
+
+ENV PYTHONDONTWRITEBYTECODE=1 \
+    PYTHONUNBUFFERED=1 \
+    PORT=8080
+
+WORKDIR /app
+
+COPY pyproject.toml README.md ./
+COPY src/ ./src/
+RUN pip install --no-cache-dir .
+
+# 4 workers x 8 threads: the scorer is pure CPU and sub-millisecond, so the
+# limit is request concurrency, not compute. Timeout well under the tuning
+# service's 300 s ceiling.
+CMD exec gunicorn --bind :$PORT --workers 4 --threads 8 --timeout 120 \
+    tunifolk.rewards.service:app

LICENSE

@@ -0,0 +1,15 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

Makefile

@@ -0,0 +1,64 @@
+# tuni-folk-gemini — reproducible pipeline
+PY      := .venv/bin/python
+CORPUS  ?= ../_work/build/classified.json
+GCS     ?= gs://nemri-genai-bb-tunifolk/rlft/v1
+PROJECT ?= nemri-genai-bb
+REGION  ?= us-central1
+JOB     ?= $(shell $(PY) -c "import json;print(json.load(open('artifacts/tuning/job.json'))['name'])" 2>/dev/null)
+
+.PHONY: help venv test lint watch negatives tunisianity acceptance dataset upload \
+        deploy validate launch status clean
+
+help:
+	@grep -E '^[a-z-]+:.*?## .*$$' $(MAKEFILE_LIST) | awk 'BEGIN{FS=":.*?## "}{printf "  \033[36m%-12s\033[0m %s\n",$$1,$$2}'
+
+venv: ## create the dev environment
+	python3 -m venv .venv && .venv/bin/pip install -q -e ".[dev]"
+
+test: ## run unit tests
+	$(PY) -m pytest -q
+
+lint: ## static checks
+	.venv/bin/ruff check src tests scripts
+
+# --- reward construction (run in order; each depends on the previous) --------
+
+negatives: ## generate the adversary: base-model attempts at the same prompts
+	$(PY) scripts/make_negatives.py --n 300 --n-hard 220 --n-msa 60 --corpus $(CORPUS)
+
+tunisianity: ## fit + hold-out validate the contrastive discriminator
+	$(PY) scripts/train_tunisianity.py --corpus $(CORPUS)
+
+acceptance: ## GATE: reward must separate corpus from base-model output
+	$(PY) scripts/acceptance.py --corpus $(CORPUS)
+
+# --- dataset and launch -------------------------------------------------------
+
+dataset: ## build train/validation JSONL
+	$(PY) scripts/build_dataset.py --corpus $(CORPUS)
+
+upload: dataset ## push the dataset to GCS
+	gcloud storage cp artifacts/dataset/train.jsonl artifacts/dataset/validation.jsonl $(GCS)/ --project $(PROJECT)
+
+deploy: ## deploy the Cloud Run reward scorer
+	gcloud run deploy tunifolk-reward --source . --project $(PROJECT) --region $(REGION) \
+	  --no-allow-unauthenticated --memory 2Gi --cpu 1 --concurrency 20 \
+	  --min-instances 1 --max-instances 30 --timeout 120 --quiet
+
+validate: ## validate the reward config against the tuning API (no job created)
+	$(PY) scripts/launch_tuning.py --dry-run
+
+launch: acceptance ## acceptance gate, validate, then create the tuning job
+	$(PY) scripts/launch_tuning.py
+
+# --- monitoring ---------------------------------------------------------------
+
+status: ## report the current job's step, reward curves and checkpoints
+	$(PY) scripts/monitor.py --once --job $(JOB)
+
+watch: ## poll the current job until it terminates
+	$(PY) scripts/monitor.py --job $(JOB)
+
+clean: ## remove caches
+	rm -rf .pytest_cache .ruff_cache
+	find . -name __pycache__ -type d -prune -exec rm -rf {} +

README.md

@@ -0,0 +1,185 @@
+# tuni-folk-gemini
+
+**Reinforcement-learning fine-tuning of Gemini 3.5 Flash to compose Tunisian folk poetry
+(الشعر الشعبي التونسي / الملحون).**
+
+Tunisian folk poetry is an unusually good RLFT target. Its metrical system is *not*
+Khalilian — Muhammad al-Marzuqi's argument, adopted by Muhyi al-Din Khrayyef and by the
+editorial committee of the national corpus, is that the Tunisian dialect has `asbab` and
+`awtad` but no `fawasil`, so classical `buhur` cannot be applied at all:
+
+> «الشعر الشعبي الحديث لا يمكن أن نطبّق عليه البحور القديمة… ولا يمكن أن نضبط موازينه
+> إلا بواسطة الإيقاع» — المرزوقي، *الأدب الشعبي*، ص 83
+
+What *does* hold a poem together is a **rhyme topology** that differs per form and is fully
+decidable by a program. That gives reinforcement learning something rare: an objective,
+cheap, hard-to-game structural signal, on a task where supervised fine-tuning could only
+teach the model to imitate 843 specific poems.
+
+---
+
+## Approach in one page
+
+RLFT is worth using when you can **score** an output better than you can **demonstrate**
+one. This repository is built around making that true for a creative task:
+
+1. **The corpus supplies tasks, not targets.** RLFT examples carry no reference answer —
+   only a prompt and a `references` map the reward reads. The corpus is used to derive
+   realistic task specifications (form + `gharaḍ`, optionally a required opening
+   hemistich) and a fingerprint of the source poem. It is never a target output.
+
+2. **The reward is mostly computation, partly judgement.** A deterministic scorer
+   (weight 0.7) measures what is decidable — rhyme topology, dialect fidelity, novelty,
+   repetition. A Gemini judge (weight 0.3) covers only the residue that code cannot see.
+   They stay separate so the two curves can be watched diverging.
+
+3. **The adversary is the policy itself.** Every component is calibrated against 520
+   base-model attempts at the same prompts — never line-shuffled or cross-form text, which
+   any metric separates trivially.
+
+4. **Components have three different jobs.** One discriminates, two are saturating
+   constraints, two are guards. Holding all five to one criterion is a category error, and
+   is the single most important thing to understand about this reward — see below.
+
+5. **The task is known to be satisfiable.** Only ~41% of corpus poems labelled `malzuma`
+   actually exhibit the `ruju'` that defines the form. Prompts are drawn only from poems
+   that pass their own validator, so the policy is never asked for a structure most
+   genuine examples fail.
+
+6. **Offline measurement and the training reward are the same code.** The Cloud Run
+   service imports the same installed package the unit tests exercise.
+
+### Why some components score *below* chance, on purpose
+
+The base model **out-regularises the tradition**. Asked for `aaaB` quatrains it emits a
+textbook rhyme scheme; the oral corpus is irregular, with transmission noise and uneven
+strophes. Measured against 520 base-model generations:
+
+| Component | AUC vs base | Role | How it is verified |
+|---|---:|---|---|
+| `tunisianity` | **1.000** | discriminator | AUC ≥ 0.60 |
+| `form` | 0.320 | constraint | scrambling a real poem must lower it (0.923 → 0.513) |
+| `novelty` | 0.500 | guard | submitting the source verbatim must collapse it (→ 0.000) |
+| `non_repetition` | 0.397 | guard | a repeated line must collapse it (→ 0.086) |
+| `meter` | 0.268 | diagnostic | weight 0; reported, not optimised |
+
+A monotonic structural reward would push the policy *away* from the target register, so
+`form` and `meter` **saturate** at per-form corpus medians: full credit for reaching the
+tradition's level, nothing for exceeding it. Deleting them is not the alternative — nothing
+would then require the requested wazn.
+
+Composite held-out AUC: **0.9978** vs base darija, **0.9978** vs hard few-shot negatives,
+**0.9914** vs MSA (an axis it was never trained on).
+
+---
+
+## The four forms
+
+Every structural component derives from the rhyme topology of the four `usul`:
+
+| Form | Topology | Defining constraint |
+|---|---|---|
+| **القسيم** `qasim` | `a B / a B / a B …` | two parallel monorhymes, no `tali'`, no return |
+| **الملزومة** `malzuma` | `AA ‖ bbb A / ccc A …` | every strophe **returns** (`ruju'`) to the `tali'` rhyme — this is what *malzūma* means |
+| **الموقف** `mawqif` | `aaaB / cccB …` | fourth `ghusn` holds one rhyme fixed across the whole poem |
+| **المسدّسة** `musaddas` | `AAA ‖ bbbb AA …` | sextets whose last two `aghsan` return to the `tali'` |
+
+Implemented in [`src/tunifolk/prosody/forms.py`](src/tunifolk/prosody/forms.py).
+
+---
+
+## Current run
+
+| | |
+|---|---|
+| Job | `tunifolk-rlft-v5` · `projects/808513141082/locations/us-central1/tuningJobs/5782572726988308480` |
+| Base model | `gemini-3.5-flash` |
+| Training / validation | 843 / 174 prompts, stratified by form |
+| Adversary | 520 base-model darija poems (300 zero-shot + 220 few-shot) + 55 MSA |
+| Reward | composite: 0.7 × deterministic structural scorer + 0.3 × `gemini-3.5-flash` judge |
+| Total steps | 156 = (843 // 32) × 6 epochs |
+
+`make status` reports the live step and reward curves. See
+[`docs/06-results.md`](docs/06-results.md).
+
+---
+
+## Layout
+
+```
+src/tunifolk/
+├── prosody/          the measurable core — all of it unit-tested
+│   ├── normalize.py    orthographic normalisation (never "corrects" dialect to MSA)
+│   ├── rhyme.py        rawiyy extraction + rhyme agreement
+│   ├── forms.py        the four usul as structural validators
+│   ├── meter.py        positional length regularity (a prior, not a scansion)
+│   ├── tunisianity.py  contrastive char-4gram LLR vs the base model
+│   └── novelty.py      anti-plagiarism fingerprints (hash-sampled)
+├── rewards/
+│   ├── structural.py   the single scoring entrypoint (tests == training)
+│   ├── service.py      Cloud Run reward server
+│   └── autorater.py    Gemini-as-judge config + composite builder
+├── data/               corpus loader, RLFT dataset builder
+└── tuning/client.py    typed v1beta1 tuningJobs client, incl. metrics reads
+```
+
+---
+
+## Reproducing
+
+```bash
+make venv                       # dev environment
+make test                       # 71 unit tests
+make negatives CORPUS=/path/classified.json   # generate the adversary
+make tunisianity CORPUS=...     # fit + hold-out validate the discriminator
+make acceptance CORPUS=...      # GATE: reward must beat the base model
+make dataset CORPUS=...         # build train/validation JSONL
+make upload                     # push to GCS
+make deploy                     # deploy the Cloud Run reward scorer
+make validate                   # validate the reward via the tuning API
+make launch                     # acceptance gate, then create the job
+make status                     # live step, curves, health, checkpoints
+```
+
+`make validate` is not optional in spirit: `validateReinforcementTuningReward` catches a
+broken scorer in seconds, and >80% errored reward invocations aborts a whole run.
+
+### Prerequisites
+
+- A GCP project with `aiplatform`, `run`, `cloudbuild`, `artifactregistry` enabled.
+- `roles/run.invoker` for `service-<PROJECT_NUMBER>@gcp-sa-vertex-tune.iam.gserviceaccount.com`
+  on the reward service — the tuning agent is the caller.
+- The corpus `classified.json`. **Not vendored** — see
+  [`docs/02-dataset-design.md`](docs/02-dataset-design.md) for provenance.
+
+---
+
+## Documentation
+
+| | |
+|---|---|
+| [01 — Background](docs/01-background.md) | the tradition, the forms, the sources |
+| [02 — Dataset design](docs/02-dataset-design.md) | prompt construction, stratification, provenance |
+| [03 — Reward design](docs/03-reward-design.md) | **each component, its role, and what the reward cannot see** |
+| [05 — Running a job](docs/05-running-a-tuning-job.md) | hyperparameters, monitoring, the API's sharp edges |
+| [06 — Results](docs/06-results.md) | protocol, pre-registered failure conditions, telemetry |
+
+## What ships
+
+The RLFT datasets **are** committed: `artifacts/dataset/train.jsonl` and
+`validation.jsonl` are the exact bytes uploaded to GCS and consumed by the tuning job, so a
+run is reproducible from this repository alone. So are the negatives, the fitted
+discriminator and the reward-gate reports.
+
+No corpus text is committed. The corpus is supplied at runtime by path
+(`CORPUS=/path/to/classified.json`) and lives outside this repository. The prompts carry
+only a first hemistich and a hash fingerprint of their source poem, never the poem.
+
+The one derived artifact that ships is a 300 KB contrastive n-gram model. Chaining its
+4-grams back into running text yields gibberish, so poems are not extractable from it;
+20/26 of a given line's 4-grams are present, which makes it a membership oracle rather than
+an extraction oracle.
+
+## Licence
+
+Apache-2.0 for the code in this repository.

artifacts/.gitkeep

@@ -0,0 +1,38 @@
+{
+  "train": {
+    "path": "artifacts/dataset/train.jsonl",
+    "examples": 843,
+    "size_mb": 2.442,
+    "max_example_chars": 8353,
+    "by_form": {
+      "qasim": 330,
+      "musaddas": 20,
+      "mawqif": 62,
+      "malzuma": 431
+    },
+    "by_mode": {
+      "compose": 402,
+      "continue": 441
+    }
+  },
+  "validation": {
+    "path": "artifacts/dataset/validation.jsonl",
+    "examples": 174,
+    "size_mb": 0.5,
+    "max_example_chars": 5311,
+    "by_form": {
+      "qasim": 72,
+      "malzuma": 94,
+      "mawqif": 6,
+      "musaddas": 2
+    },
+    "by_mode": {
+      "compose": 87,
+      "continue": 87
+    }
+  },
+  "config": {
+    "min_score": 0.6,
+    "seed": 20260731
+  }
+}