Skip to content

Commit 41c97e8

Browse files
committed
Measure reply latency and bound unstarted replies
Nothing said where the wait between a candidate finishing and Jim answering went in enough detail to act on. Beside the per-turn turn_timing summary, the agent now logs one structured record per reply, from the candidate's last voiced frame or the prompt that asked for it to the first audio frame handed to LiveKit: endpointing, the provider's first output, tool rounds, output queueing and context size, with provider milestones stamped when the socket received them. scripts/analyze-latency.py partitions each reply into consecutive segments and reports the dominant one. It keeps replies past --stall-ms out of the distributions and counts them as a rate, splits replies by whether a tool round came before their audio, fits growth with context within each kind of reply, and marks statistics over small samples. A socket that hangs sends nothing at all, yet a reply that never started waited out the same 45 seconds as one that started and stalled. A reply nothing has started toward is now replaced after CODETRIAL_GEMINI_FIRST_OUTPUT_S, ten seconds by default and never more than the configured reply timeout, unless Gemini reports the candidate still in their turn. A recovery, a replacement's briefing or a reply check, is never held to less than that default while it is the newest request, so a shorter setting cannot cut one recovery straight into the next. A provider interruption no longer settles a reply nobody heard, and a turn that answered a waiting candidate with tool calls alone gets one note asking for the reply if one is due. The reply path loses its own delays: the agent-state attribute write moves to a background writer, microphone audio goes to Gemini in 20 ms batches, and a hold always cuts a generation in progress. The note after a tool-only turn is live prompt 27 (bundle 36).
1 parent b491026 commit 41c97e8

32 files changed

Lines changed: 5705 additions & 227 deletions

‎.cargo/mutants.toml‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
# Functions the mutation gate cannot judge, because `cargo test` cannot reach
22
# them. Matched against the mutant names that `cargo mutants --list` prints.
33
#
4-
# EXCLUSIONS: 69
4+
# EXCLUSIONS: 71
55
#
66
# That number is checked by `scripts/test.sh`, so adding an entry means editing
77
# this line too. The point is not the count, it is that the list only ever grows
@@ -299,6 +299,11 @@
299299
# builds the Google URL for `generate_phase_judgment_at`, which is tested
300300
# against a local server; reaching the real one needs a live key.
301301
#
302+
# `log` in the latency module and `log_first_frame` beside the output worker
303+
# only write a line to stderr; `()` leaves nothing a test can read. What the
304+
# line holds is built and tested apart from the printing: the trace's own
305+
# records, and `first_frame_record`.
306+
#
302307
# Keep this list short and each entry justified. An entry that is really "we
303308
# never got around to testing this" belongs in a test, not here.
304309
exclude_re = [
@@ -371,4 +376,6 @@ exclude_re = [
371376
"settle_phase_judge",
372377
"flush_framework_progress",
373378
"generate_phase_judgment_with_keys",
379+
"livekit/latency\\.rs:.*replace log with \\(\\)$",
380+
"replace log_first_frame with \\(\\)$",
374381
]

‎README.md‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -264,6 +264,8 @@ The gate ends by printing what it did not check, because several lanes are
264264
optional and a run that skips them still exits 0. That summary, the formatter
265265
chain, the git hooks, the generated problem files, and where a slow run's time
266266
actually goes are in [docs/development.md](docs/development.md).
267+
[Reply latency observability](docs/latency-observability.md) explains per-turn
268+
server timing, tool execution and audio queue breakdowns.
267269

268270
## Configuration
269271

@@ -278,7 +280,8 @@ The common ones:
278280
| `GEMINI_LIVE_MODEL` | `gemini-3.1-flash-live-preview` | Realtime interviewer model |
279281
| `GEMINI_REPORT_MODEL` | `gemini-3.1-flash-lite` | Report model |
280282
| `CODETRIAL_MAX_INTERIM_REVIEWS` | `6` | Quiet-pause report-model reviews per interview; `0` disables them and `72` is the maximum |
281-
| `CODETRIAL_GEMINI_REPLY_TIMEOUT_S` | `45` | Seconds an owed interviewer reply may go without output before the Live socket is replaced (20–120); see [degradation controls](docs/provider-cost-and-degradation.md) |
283+
| `CODETRIAL_GEMINI_REPLY_TIMEOUT_S` | `45` | Seconds a started interviewer reply may go without further output before the Live socket is replaced (20–120); see [degradation controls](docs/provider-cost-and-degradation.md) |
284+
| `CODETRIAL_GEMINI_FIRST_OUTPUT_S` | `10` | Seconds an owed reply that nothing has started toward may go with nothing back before the Live socket is replaced (5–120, never above the reply timeout) |
282285
| `CODETRIAL_GEMINI_CANDIDATE_VIDEO_ENABLED` | `false` | Forward candidate video to Gemini, one low-resolution frame in five seconds; ignored in a whiteboard interview |
283286
| `CODETRIAL_COMPILER_EXPLORER_ENABLED` | `true` | Enable remote C, C++, and Java runs |
284287
| `CODETRIAL_MAX_CONCURRENT_INTERVIEWS` | `16` | Interviews one `web` process hosts agents for |

‎config/codetrial.env.example‎

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -8,10 +8,14 @@ GEMINI_REPORT_MODEL=gemini-3.1-flash-lite
88
# Reviews quiet candidate stretches with the report model. 0 reserves that
99
# quota for final reports; the default is 6 and the maximum is 72.
1010
# CODETRIAL_MAX_INTERIM_REVIEWS=6
11-
# Seconds an owed interviewer reply may go without output before the Live
12-
# socket is replaced. Raise it if `timing:` lines show slow replies that do
11+
# Seconds a started interviewer reply may go without further output before the
12+
# Live socket is replaced. Raise it if `timing:` lines show slow replies that do
1313
# arrive; the default is 45, bounded to 20-120.
1414
# CODETRIAL_GEMINI_REPLY_TIMEOUT_S=45
15+
# Seconds an owed reply nothing has started toward may go with nothing back
16+
# before the socket is replaced. Raise it for a model whose replies are slow to
17+
# begin; the default is 10, bounded to 5-120 and never above the reply timeout.
18+
# CODETRIAL_GEMINI_FIRST_OUTPUT_S=10
1519
GEMINI_VOICE=Puck
1620
CODETRIAL_ROOM_PREFIX=interview
1721
CODETRIAL_DURATION_MIN=45

‎docs/interview-contract-versions.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,10 +8,11 @@ can select it.
88

99
## The active bundle
1010

11-
Bundle 35: live prompt 26, report prompt 20, rubric 1, report schema 3.
11+
Bundle 36: live prompt 27, report prompt 20, rubric 1, report schema 3.
1212

1313
| Bundle | Introduced |
1414
|---|---|
15+
| 36 | A turn that answered the candidate with tool calls alone and said nothing is followed by one note that asks for the reply if one is due; Jim may still answer it with silence, but not with nothing, since the note is owed a completed turn. The report prompt, rubric and report schema are unchanged. |
1516
| 35 | The live interviewer receives the same worked examples shown on the candidate's screen, generated from the problem bank. When the candidate hides examples, their contents are omitted from the live prompt. Live prompt 26; the report prompt, rubric and report schema are unchanged. |
1617
| 34 | Optional English accent directions are appended by the live prompt builder without changing the original interview instructions. Default adds no text. Voice and accent Random choices are resolved independently before token issuance and retained for the interview and reconnects. Live prompt 25; report prompt 20, rubric 1 and report schema 3 are unchanged from bundle 33. The server stamps the resolved `interviewerVoice` and `interviewerAccent` after report generation, including incomplete reports. Default voice records the configured Gemini voice; Default accent records Default. These optional provenance fields are normalized at the browser sanitizer, following bundle 7, without changing scores or existing field semantics. Reports without them remain readable without inventing missing interviewer settings. |
1718
| 33 | The live clock policy explicitly forbids reading the TIMER label or its sentence aloud. The last platform reading remains authoritative; requested time answers and the five-minute warning retain their existing rules. Tool-assisted totals remain spend counters and never seed the comparable context peak. A total below the last completed single-generation prompt can establish a decrease; all observations in the tool turn are considered before deciding. Provisional usage received before a tool call is retracted from the comparable peak. Live instructions shorten repeated speech, evidence and interview-step guidance while retaining the three-sentence limit, safety boundaries, evidence requirements and mode-specific behavior. A paged `read_conversation` tool retrieves recorded candidate and interviewer speech omitted from local recovery tails, with substring search, byte cursors, bounded serialized pages and no platform instructions or private review notes. Recovery directs missing earlier details to that tool. Report brief 20 removes the rolling-assessment direction to report embedded instructions. Editor and board material require an exact candidate excerpt before reporting an attempted instruction; fences, headings and platform closing events do not establish one, and feedback does not describe prompt handling. Recovery removes the quoted owed event's superseded final platform clock. Additional anti-script wording was tested and withdrawn: it increased prompt tokens without preventing spoken timer leakage. Configured continuity inputs move the event clock to the final position instead of duplicating it. Configured editor, board and hint tool replies also move their redundant result clock to the authoritative continuity clock. Evidence-recording continuity omits the clock, matching ordinary evidence replies and retaining platform/read-tool time sources. A tool-response batch carries continuity and the final framework checklist once each in their final eligible responses, preserving every result and ordered state change. The final eligible coding evidence response can carry a changed, parsed editor excerpt once so speech-deferred reviews do not leave a stale editor for the tool continuation. A successful changed-code response, including a default editor read, requests the proactive review once using the final batch state, prioritizing the candidate's latest question; a default read that stops at its page limit neither carries the review nor marks the buffer shown, and only a read showing the whole buffer marks it shown, so later-line pages and requested hints whose editor page stopped at its limit do not; only a delivered review consumes an unreviewed parsed substantive revision, so explicit later-line reads, failed writes and later edits still arm the watcher. Timed replies send their private clock as non-generating ordered context, followed by the action on the same ordered channel; while the provider owes or is producing a reply, the action goes as realtime text instead, since a completed client turn would interrupt that reply even before its first output. The socket counts as owing one from detected candidate speech, its transcription, an interruption, model output or a reply-requesting send until the turn completes, and provider evidence lapses after 10 seconds while a requested reply stays owed through the longest configurable reply deadline, which the reply watchdog enforces first. Each timed reply counts as one input. Silent context and untimed input retain their transport. Timer-like candidate source text stays intact. Provider speech activity, while audio is attached and frames are not paused, defers idle prompts, checkpoint refreshes and idle assessment calls until speech ends, without treating activity as transcript evidence. Advisory transport restarts wait through the quiet transcription tail before resuming a stale checkpoint. During ongoing speech the periodic room watch checks held advisories without waiting for another provider event or stall; they finalize the partial transcript before replacing the socket, cancel its unheard reply, and restore the full quoted prefix silently before queued audio continues. The silent speech handoff omits its clock; requested-time reads retain their clock sources. A bounded 40-second microphone queue retains frames while the room loop reconnects, replacing the dependency default of 100 ms. Only a post-cancellation checkpoint can resume that handoff; a resumed socket receives the listening note alone and keeps its context baseline, and a cold one the full cold briefing before it. A platform request the replaced socket still owed is named in the note and stays owed. Events the replaced socket emits while the prefix is finalized are counted in the log. Failed writes retain local briefing debt. Coding-only Live tool declarations offer only the six REACTO phases; behavioral sessions retain all ten phase choices. Tool descriptions remove procedural rules already present in the system instructions while retaining unique paging and snapshot semantics. Whiteboard compression checkpoints omit editor and test data and request the current drawing through `read_board`; a behavioral checkpoint does not return to the drawing. Report repairs compact only whitespace outside JSON strings in complete bounded responses, preserving values and existing malformed or oversized fallback behavior. Whiteboard initial and recovery instructions describe drawing and evidence rather than editing, implementing a function or passing executed tests; coding instructions remain unchanged. Editor and requested-hint reads distinguish a stale or unknown browser-report source from current code, independently of retained Test credit; current and no-report reads remain unchanged. Editor-update outcomes, combined-mode progress and recovery also use exact displayed-report provenance while retaining edit-tolerant Test credit. Recovery briefings add only the owed-question rule and any opening or latest candidate line their transcript tail no longer holds whole, rather than repeating their own no-restart rules or asking a silent checkpoint to continue. Tool continuity follows the same plausibility gate and is omitted while the floor is held. Configured compression carries bounded continuity beside platform inputs after the opening while a cut is plausible (before any measurement, after a detected cut, or while the last comparable prompt count is in the upper half of the band between target and trigger), including the active round, evidenced steps, pending request and a bounded prior candidate opening in coding, so the first response after a context cut can continue without replaying the greeting or full problem restatement. Provider-default sessions retain their existing input path. Default editor reads can collapse identical consecutive leading comments while keeping every distinct note, original line numbers and explicit line access. Tool-assisted usage sums remain spend counters but are excluded from context-size comparisons to avoid false checkpoints. Completed coding retains the first silence check and 30-second attempt throttle, then doubles the spacing of successful same-episode nudges up to 480 seconds. Meaningful candidate activity resets the episode; failed writes do not advance its successful-send cadence. Suppressing a nudge still permits significant editor reviews, and unfinished coding, behavioral support, required replies and trusted timers retain their behavior. Active provider speech detection plus recent voiced microphone frames defers reply-timeout, generation-stall and deferred advisory restarts across watch, event and playout paths. Speech end or acoustic quiet rebases outstanding reply clocks once; stale activity edges alone cannot renew them. A provider speech start with no microphone or provider evidence for 10 seconds is treated as ended. Paused frames do not count as acoustic evidence, context resets discard the acoustic history, and the absolute interview deadline remains authoritative. During real incoming speech, a parsed substantive editor change settled for ten seconds can be sent as non-generating context. Successful delivery has its own language-qualified snapshot and becomes the baseline later editor updates diff from, without consuming optional review or reply debt; silent delivery itself diffs from the code already shown by any route and sends nothing when the model already holds the current code; failed writes remain pending and socket recovery resets or initializes that snapshot. Ordered local writes do not guarantee consumption order against realtime audio. Coding transitions are conditional on implementation not having begun; otherwise review current work. Recovery, review, silence and timer prompts can carry a bounded edits-presence note, until Coding is evidenced, without treating edits as a completed or correct implementation. Silence nudges omit the empty-editor branch when the editor holds candidate edits, and coding-only sessions omit the behavioral-question ban from nudges and reviews, which their system instruction already carries. Silence policy still selects its existing setup-error and next-step branches before that note. Existing final checklist tool responses can expose candidateEditsPresent only in the coding editor round, using the unchanged written-content heuristic; a comment or stub may qualify. Tool continuity wording is shorter with identical latest-utterance selection, trust fences, JSON data and timer. Cold and checkpoint recovery envelopes are shorter while retaining archive lookup before earlier-speech answers, reconciliation with the current local record, historical-data trust boundaries and the active-session and no-restart rules. Completed Example walkthroughs are not reopened without a relevant implementation change or a concrete unresolved concern; initial ordinary and boundary examples and execution-mode Test requirements are preserved. Outside candidate speech, judged from the microphone's own voice level or provider speech detection, each Live socket receives one camera presence frame; frames during speech keep the five-second interval, and board images are unaffected. |

0 commit comments

Comments
 (0)