Skip to content

Commit 65b8e4d

Browse files
committed
Take evidence recording off the editor reply path
Two in three spoken replies still waited on a tool round, and nearly every one was the interviewer calling record_framework_evidence for Repeat, Example or Algorithm: steps the prompt told it to leave alone and the phase judge records anyway. Each call holds the reply back by a second model pass. Neither the prompt nor leaving those steps out of the tool's schema stopped the calls, so at a typed editor the tool is no longer declared. The prompt says the platform records each step, and that the editor is not to be read to learn whether one was: without the tool's replies the model did that instead, a tool round all the same. Test events there no longer ask for an evidence call either. The phase judge now covers what only the tool recorded there. It records Test from a hand trace while nothing can run the written code, judged with the code in view and only from speech, and the STAR steps, quoted only from inside the behavioral round and not from a story the interviewer told first; it reads the round as new once it holds something, and again when its start moves back to the interviewer turn in flight. Repeat, Example and Algorithm stay open after Coding while a step about the code is, and a coding-round step may be quoted from a comment the candidate typed, as one whose speech was not recognized is asked to, recorded as an editor snapshot. It can report a declined probe, and the platform writes the end-of-session skips for a started round that the wrap-up used to ask for, except where the probe was declined. The coding round's routine judgments leave 8 of the 48 for a behavioral round still to come; the last judgment and a round transition may spend them. A whiteboard keeps the tool, its wording and the judge as they were, since the judge is not shown the board.
1 parent cfe38ad commit 65b8e4d

14 files changed

Lines changed: 967 additions & 314 deletions

File tree

‎docs/interview-contract-versions.md‎

Lines changed: 1 addition & 1 deletion
Large diffs are not rendered by default.

‎docs/provider-cost-and-degradation.md‎

Lines changed: 15 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -195,21 +195,26 @@ rate. `CODETRIAL_MAX_INTERIM_REVIEWS` defaults to 6, accepts `0` to disable
195195
the reviews, and is capped at 72. This is a quota guard, not a completeness
196196
limit: the final report still receives the complete transcript and editor state.
197197

198-
The phase judge, which records the conversational REACTO steps from the
199-
candidate's own words, uses the same report model and keys. Every judgment
200-
needs a candidate turn or editor it has not read, or the window of a call that
201-
failed, and a step it records still open: Repeat, Example and Algorithm until
202-
Coding is recorded, Coding and Optimizations while there is code. On the watch
198+
The phase judge, which records the interview's steps from the candidate's own
199+
words and editor, uses the same report model and keys. At a typed editor it is
200+
their only recorder, since the interviewer has no evidence tool there. Every
201+
judgment needs a candidate turn or editor it has not read, or the window of a
202+
call that failed, and a step it records still open: Repeat, Example and
203+
Algorithm until Coding is recorded, and at a typed editor after it while a step
204+
about the code is, Coding and Optimizations while there is code, and at a typed
205+
editor Test while nothing can run the written code and the STAR steps once the
206+
behavioral round has opened, read from where it opened. On the watch
203207
tick one starts once the candidate has been quiet and not typing for 2 seconds,
204208
no more often than every 6 seconds, or every 30 seconds when only the editor
205209
changed; typing pauses every few seconds while Coding is open, so editor-only
206210
judgments also take at most 12 of the calls. The end and a held round
207211
transition may start one sooner: the end waits at most 5 seconds in all for its
208212
judgments, and a round transition is held at most 5 seconds while the room
209213
keeps running, or until the resume when the interview is paused. Both share the
210-
ceiling of 48 calls per interview. Each reads at most 6 KiB of transcript tail
211-
and 6 KiB of editor. A rate limit on a judgment does not cool the key, so the
212-
judge cannot leave the final report without one.
214+
ceiling of 48 calls per interview, of which a coding round at a typed editor
215+
leaves 8 for a behavioral round still to come. Each reads at most 6 KiB of
216+
transcript tail and 6 KiB of editor. A rate limit on a judgment does not cool
217+
the key, so the judge cannot leave the final report without one.
213218

214219
## What bounds concurrency
215220

@@ -408,8 +413,8 @@ and prohibit returning to it merely to recover earlier work. They also omit
408413
coding-run progress cues. Coding checkpoint output is unchanged by these mode
409414
guards; no token reduction is claimed for this correctness fix.
410415

411-
Only while a pair is configured do `read_editor`, `log_hint` and
412-
`record_framework_evidence` answers also carry the latest unanswered candidate
416+
Only while a pair is configured do `read_editor`, `log_hint` and, at a
417+
whiteboard, `record_framework_evidence` answers also carry the latest unanswered candidate
413418
utterance, as quoted data, so a reply owed across a cut is not lost, and only
414419
while a cut is plausible (as below) and the floor is not held. Without a
415420
pair these application-level additions are omitted. Provider-default sliding

‎src/agent.rs‎

Lines changed: 212 additions & 53 deletions
Large diffs are not rendered by default.

‎src/agent/events.rs‎

Lines changed: 22 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -622,7 +622,11 @@ fn apply_test_results(
622622
};
623623

624624
if let Some(evidence) = recorded {
625-
if let Some(reminder) = super::unrecorded_earlier_phases(state, &evidence) {
625+
// The reminder asks for an evidence call, which an editor without the
626+
// tool cannot make; the judge records what the candidate did.
627+
if super::evidence_tool_declared(state.interview_mode)
628+
&& let Some(reminder) = super::unrecorded_earlier_phases(state, &evidence)
629+
{
626630
reply.push('\n');
627631
reply.push_str(&reminder);
628632
}
@@ -819,6 +823,13 @@ fn control_round_transition(
819823
state.behavioral_round_prior_turn =
820824
super::last_speaker_line(&state.transcript, INTERVIEWER_SPEAKER)
821825
.map(|(index, line)| (index, line.clone()));
826+
827+
// The judge reads the round from where it opens, which is here until
828+
// the interviewer turn in flight at this packet is seen to change and
829+
// moves it back: an answer the coding round's judgments read is then
830+
// judged again for STAR, because it is inside the round.
831+
state.judged_round_from = state.transcript.len();
832+
state.judged_last_turn.clear();
822833
state
823834
.evidence_ledger
824835
.record_lifecycle(receipt_timestamp_ms, LifecycleTransition::BehavioralStarted);
@@ -830,7 +841,7 @@ fn control_round_transition(
830841
} else {
831842
DataEventResult {
832843
round_changed: Some("skipped"),
833-
generate_reply: Some(round_skipped()),
844+
generate_reply: Some(round_skipped(state.interview_mode)),
834845
..DataEventResult::default()
835846
}
836847
}
@@ -878,10 +889,15 @@ fn control_end_interview(
878889
payload: &serde_json::Value,
879890
receipt_timestamp_ms: u64,
880891
) -> DataEventResult {
881-
// Only for a round that never opened. Once it has, a step left without a
882-
// row may be one the candidate declined, which stays unassessed rather than
883-
// skipped, and only the conversation can tell the two apart.
884-
if !state.behavioral_round_started {
892+
// A round that never opened is closed here. Once it has, a step left
893+
// without a row may be one the candidate declined, which stays unassessed
894+
// rather than skipped, and only the conversation can tell the two apart. An
895+
// interviewer holding the evidence tool reads it and records the skips at
896+
// the wrap-up. Without the tool the phase judge has read it: the end waits
897+
// for its last judgment, which says whether the probe was declined.
898+
if !state.behavioral_round_started
899+
|| (!super::evidence_tool_declared(state.interview_mode) && !state.star_declined)
900+
{
885901
super::skip_unassessed_star(state, "The session ended before assessment.");
886902
}
887903
let prior_code = state.code.clone();

0 commit comments

Comments
 (0)