Run the behaviour check through the shim

The scripted behaviour check and the live phase-judge test posted to
Google's URL by hand, so neither could follow CODETRIAL_GEMINI_REST_BASE
the way the report does. Both now build the report's URL, so pointed at
scripts/gemini-shim.py they send what production sends, tools included,
and the judge test prints each judgment it applies. The played
candidates keep BEHAVIOR_LOCAL_BASE for talking to an OpenAI-compatible
server directly. Against gemma-4-12b with --thinking off the check takes
about four minutes, and what it fails on is the model's behaviour,
chiefly a hint request answered without log_hint, which is what the
results collected in #257 compare.

Author: alanhc

README.md

@@ -317,6 +317,14 @@ Gemini. Any base other than Google's gets longer report deadlines, 45 seconds
 a call and 250 in all instead of 20 and 125, since a 12B model on one 16 GB GPU
 takes 14 to 32 seconds per report.
 
+The shim carries function calls too, so the interviewer behaviour check in
+[docs/development.md](docs/development.md#checks-outside-the-gate) runs against
+the same base. With Gemma 4, start it with `--thinking off`. That check names no
+thinking budget, and with thinking on gemma-4-12b sometimes repeated itself to
+the output limit or put its reasoning in the reply. With it off the check
+passed 18 of 21 problem runs across seven runs, about 24 seconds a run against
+240; each miss was a second hint request answered without calling `log_hint`.
+
 ## Recording
 
 Recording is off by default. Enabling it requires a separate LiveKit project,

docs/development.md

@@ -238,11 +238,11 @@ reads them.
 ## Checks outside the gate
 
 Whether the interviewer actually follows the live prompt, rather than whether
-the prompt says the right things, needs a Gemini key. The check scripts a
-candidate through three problems against a text model given the same
-instructions, greeting and tools, and fails on a named source, a volunteered
-limit, an unanswered size question, or a hint that goes past the rung it was
-served:
+the prompt says the right things, needs a Gemini key or a local model. The
+check scripts a candidate through three problems against a text model given
+the same instructions, greeting and tools, and fails on a named source, a
+volunteered limit, an unanswered size question, or a hint that goes past the
+rung it was served:
 
 ```bash
 scripts/interview-behavior-check.sh
@@ -252,6 +252,19 @@ BEHAVIOR_PROBLEMS=3sum,lru-cache scripts/interview-behavior-check.sh
 A free key allows fifteen requests a minute, so the check waits out rate
 limits; three problems take about two minutes.
 
+Against a local model, point it at `scripts/gemini-shim.py` the way the report
+is pointed, so the check sends what production sends through the shim the
+report uses. The key is still read but goes no further than the shim, which
+ignores it, so any non-empty value does when the config file has none:
+
+```bash
+CODETRIAL_GEMINI_REST_BASE=http://127.0.0.1:8090 GOOGLE_API_KEY=local \
+    scripts/interview-behavior-check.sh
+```
+
+`BEHAVIOR_LOCAL_BASE` instead talks to an OpenAI-compatible server directly,
+without the shim; it is what the played candidates use for their own side.
+
 The end-to-end browser check additionally needs Playwright and Chromium:
 
 ```bash

src/gemini.rs

@@ -2065,7 +2065,7 @@ fn tool_ack_pcm(
 /// No `?key=` here on purpose. A `reqwest` error Displays the URL it was built
 /// from, and this call's errors reach the candidate's browser in the report
 /// failure note, so the credential travels in a header instead.
-fn gemini_generate_content_url(model: &str) -> String {
+pub fn gemini_generate_content_url(model: &str) -> String {
     format!(
         "{}/v1beta/models/{}:generateContent",
         rest_base(),

tests/interview_behavior.rs

@@ -240,9 +240,14 @@ fn the_rules_catch_a_named_source_and_a_volunteered_limit() {
 }
 
 /// Who plays the interviewer. Gemini is the model production runs; a local
-/// OpenAI-compatible server (llama.cpp's `llama-server`) plays it for free,
-/// which makes it a fair stand-in for comparing candidates against one
-/// another and a poor one for judging Gemini.
+/// model plays it for free, which makes it a fair stand-in for comparing
+/// candidates against one another and a poor one for judging Gemini.
+///
+/// A local model is reached one of two ways. `Gemini` follows
+/// `CODETRIAL_GEMINI_REST_BASE` the way the report does, so pointed at
+/// `scripts/gemini-shim.py` it exercises the same request shape and the same
+/// shim the report uses. `Local` skips both and talks to an OpenAI-compatible
+/// server directly, translating here.
 #[derive(Clone)]
 enum Backend {
     Gemini { key: String, model: String },
@@ -396,10 +401,7 @@ impl Conversation {
         for _ in 0..8 {
             let response: Value = self
                 .client
-                .post(format!(
-                    "https://generativelanguage.googleapis.com/v1beta/models/{}:generateContent",
-                    model
-                ))
+                .post(codetrial::gemini::gemini_generate_content_url(model))
                 .header("x-goog-api-key", key)
                 .json(&json!({
                     "systemInstruction": { "parts": [{ "text": self.instructions }] },
@@ -1617,9 +1619,7 @@ async fn judge_phases(state: &mut RuntimeState, problem: &'static Problem) -> Ve
         .filter(|model| !model.is_empty())
         .unwrap_or_else(|| codetrial::config::DEFAULT_GEMINI_REPORT_MODEL.to_string());
     let response: Value = reqwest::Client::new()
-        .post(format!(
-            "https://generativelanguage.googleapis.com/v1beta/models/{model}:generateContent"
-        ))
+        .post(codetrial::gemini::gemini_generate_content_url(&model))
         .header("x-goog-api-key", gemini_key())
         .json(&json!({
             "systemInstruction": { "parts": [{ "text": phase_judge_system_instruction() }] },
@@ -1632,13 +1632,14 @@ async fn judge_phases(state: &mut RuntimeState, problem: &'static Problem) -> Ve
         }))
         .send()
         .await
-        .expect("Gemini is reachable")
+        .expect("the report model is reachable")
         .json()
         .await
         .expect("Gemini answers JSON");
     let text = response["candidates"][0]["content"]["parts"][0]["text"]
         .as_str()
         .unwrap_or_else(|| panic!("no judgment: {response}"));
+    println!("judgment: {text}");
     apply_phase_judgment(state, text);
     framework_progress(state)
 }