Skip to content

08-rl-running.md

Latest commit

 

History

History
183 lines (148 loc) · 8.5 KB

08-rl-running.md

File metadata and controls

183 lines (148 loc) · 8.5 KB
Raw

05 — Running a tuning job

Operational notes for RLFT on Gemini, including the sharp edges we hit. The service is Pre-GA: do not send proprietary or confidential data, and do not use it in production.

Environment
Project YOUR_PROJECT_ID (number YOUR_PROJECT_NUMBER)
Tuning region us-central1
Serving region us multi-region (aiplatform.us.rep.googleapis.com)
API version v1beta1 only
Base model gemini-3.5-flash
Reward service https://your-reward-service.us-central1.run.app
Dataset gs://YOUR_GCS_BUCKET/rlft/v1/

RLFT runs only in us-central1 and europe-west4. v1 has no RLFT schemas at all — calling it returns a response with fields silently missing, which is a nasty way to debug. TuningClient hard-codes v1beta1 for that reason.

Sharp edges, in the order we hit them

1. The base model need not be servable in the tuning region. gemini-3.5-flash:generateContent returns 404 in us-central1 for this project — it is only served from global. Tuning accepts it anyway. Serving availability and tuning eligibility are different questions; we confirmed by creating a throwaway job with a deliberately invalid dataset URI and observing that the model was accepted.

2. validateReinforcementTuningReward lives under tuningJobs. The path is …/locations/{loc}/tuningJobs:validateReinforcementTuningReward, not …/locations/{loc}:validateReinforcementTuningReward. The latter 404s with an HTML error page.

3. Its request field is example, not reinforcementTuningExample.

4. An autorater reward needs an explicit judge model. Omitting autoraterConfig.autoraterModel yields:

Error for reward [poetic_quality]: Missing autorater model in the autorater_config

and a NaN reward. Worth stressing what this would have cost: NaN counts as an errored invocation, >80 % errored invocations aborts the job, and this reward was 30 % of the composite. Validation caught it in seconds.

5. Judge availability is project- and region-specific. gemini-3.1-pro-preview and gemini-3-pro-preview 404 in us-central1 here. Probe before configuring:

for M in gemini-2.5-pro gemini-2.5-flash gemini-3-pro-preview; do
  curl -s -o /dev/null -w "$M %{http_code}\n" -X POST \
    -H "Authorization: Bearer $(gcloud auth application-default print-access-token)" \
    "https://us-central1-aiplatform.googleapis.com/v1beta1/projects/$P/locations/us-central1/publishers/google/models/$M:generateContent" \
    -H 'Content-Type: application/json' -d '{"contents":[{"role":"user","parts":[{"text":"hi"}]}]}'
done

6. ParsedResponseConversionScorer is an empty message. It takes no properties; it is a marker meaning "convert the parsed autorater response straight to a float".

7. IAM. The tuning agent — service-<PROJECT_NUMBER>@gcp-sa-vertex-tune.iam.gserviceaccount.com — is the caller of your Cloud Run reward, so it needs roles/run.invoker. The service itself stays --no-allow-unauthenticated.

Hyperparameters, and why

batchSize: 32
samplesPerPrompt: 16
epochCount: 6
learningRateMultiplier: 1.0
adapterSize: ADAPTER_SIZE_SIXTEEN
maxOutputTokens: 32768
thinkingLevel: HIGH
evaluateInterval: 13
checkpointInterval: 26
  • batchSize × samplesPerPrompt must be divisible by 16 and ≤8,192 for text (≤1,024 for multimodal). 32 × 16 = 512, comfortably inside.
  • epochCount: 6. 843 prompts / 32 ≈ 26 steps per epoch, so ≈158 steps — well under the hard 500-step ceiling that overrides epochCount regardless of what you ask for. The service default would have been min(110 × promptsPerBatch / promptsInDataset, 10).
  • maxOutputTokens: 32768 — the model maximum and the service default. An earlier run used 4096 on the reasoning that "poems are short, so don't pay for unused length". That was wrong twice. First, the parameter is an upper bound: a generation that stops at 800 tokens costs 800 tokens regardless of the cap, so lowering it saves nothing. Second, and worse, the budget includes thinking tokens — with thinkingLevel: HIGH the model can spend an unpredictable share of it reasoning before writing a line, so a low cap silently truncates the poem. A truncated poem has a broken rhyme topology, and the policy would have been trained on mutilated samples that the reward then punished for a defect the config caused.
  • thinkingLevel: HIGH. Composing to a rhyme topology is constraint satisfaction, so deliberation should help. The narrative docs list only MINIMAL and HIGH; the API enum actually accepts MINIMAL | LOW | MEDIUM | HIGH, which is worth knowing if you want to ablate.
  • Intervals are in steps, not epochs, and checkpointInterval should be an integer multiple of evaluateInterval. 13 ≈ twice per epoch, 26 ≈ once per epoch.
  • adapterSize: ADAPTER_SIZE_SIXTEEN (default). The enum runs ONE | TWO | FOUR | EIGHT | SIXTEEN | THIRTY_TWO.

Launching

make validate   # dry-run the reward through the API — do not skip
make launch     # validate, then create

scripts/launch_tuning.py refuses to create a job unless every reward in the validation response is present, finite and inside [-1, 1] (validation_problems). The check is structural rather than a substring scan of the serialised JSON, which would both miss errors nested under unexpected keys and fire on a reward merely named error_rate.

Monitoring

GetTuningJob returns state, tunedModel, error and metadata. It carries no step and no progress field — its only step is tunedModel.checkpoints[].step, which advances once per checkpointInterval. A job can therefore look frozen for hours while training normally.

The metrics are reachable over REST, through the experiment's backing Tensorboard:

tuningJob.experiment                            -> Metadata Context
  Context.metadata.backing_tensorboard_resource -> Tensorboard
    experiments/{id}/runs/{run}/timeSeries      -> 41 series
      tensorboards/{id}:batchRead               -> scalars, with wallTime

TuningClient.metrics() implements that chain; TuningClient.current_step() reads the live step from it. Total steps are derived, not reported: tuningDatasetExampleCount // batchSize * epochCount.

make status   # one report: step, curves, health, checkpoints
make watch    # poll until the job terminates

Two cautions when reading scalars directly. ProtoJSON omits a 0.0 value entirely, so an absent scalar means zero, not missing. And the step you read is the last logged step, so the in-flight one is step + 1.

The series worth watching, given this reward design:

metric why
/train_mean_reward, /eval_mean_reward the headline; eval is the honest one
structure/train_mean_reward vs poetic_quality/train_mean_reward divergence between these two is the reward-hacking signature
/train_generation_length collapse toward the minimum = length gaming; unbounded growth = drift toward the maxOutputTokens cap, after which generations truncate
structure/train_rpc_error_ratio Cloud Run health; >0.8 aborts the job
/learnable_prompt_ratio share of samples surviving filtering
${reward}/train_clipping_rewards_ratio how often rewards hit the [-1,1] clip

Job states: JOB_STATE_PENDING → JOB_STATE_RUNNING → JOB_STATE_SUCCEEDED | FAILED | CANCELLED.

Afterwards

A successful job deploys the last checkpoint to an endpoint in the us multi-region. Intermediate checkpoints are produced at checkpointInterval and can be deployed and evaluated independently before the job finishes.

tunifolk status --job 5782572726988308480
tunifolk sample --endpoint projects/…/locations/us/endpoints/… --form malzuma

Continuous tuning is supported (SFT → RLFT and RLFT → RLFT) by adding a preTunedModel block — a sibling of reinforcementTuningSpec, not nested inside it — with tunedModelName and an optional checkpointId (latest if omitted).

Cost model

Tuning bills tokens in two phases: a sampling phase (generation — this is where samplesPerPrompt shows up) and a training phase. Inference on the tuned model is priced at 1.5× the base model from Gemini 3 onward. Evaluation during tuning is billed as batch prediction. The Cloud Run scorer and the autorater judge are billed separately, as Cloud Run requests and as Gemini calls respectively — the judge is not free, which is a second reason to keep its weight and samplingCount modest.