Extend practice feedback and problem bank checks by jserv · Pull Request #54 · sysprog21/codetrial

jserv · 2026-09-16T20:59:51Z

This branch widens what a practice session gives back and tightens what the problem bank has to prove. Candidates can write their own test cases and see what their code returns, practice without a camera when the interview is not recorded, and get a post-interview debrief that states the hint rung they reached and the level they practiced for, while the report is refused if it names the published problem. The lobby keeps a longer history whose reports can be reopened, keeps due reviews across level changes, resets a failed review to one day, and shows progress by topic. The bank now accepts original exercises, every judge needs five cases including a boundary one, and a C starter that returns an array has to say who frees it. On the server, a provider whose credentials are refused leaves rotation, and report-model reviews during quiet pauses are capped per interview. CI gains a check that the release image matches the manifest's Rust version and a weekly run that requires the pinned avatar model, and the legacy migration paths for the git hooks and vendored files are removed. The interview contract moves to bundle 8, live prompt 3 and report prompt 8, with each step recorded in docs/interview-contract-versions.md.

Verified with ./scripts/test.sh at each of the 18 commits, not only the tip, and all pass; the tip runs 672 Rust tests and 523 browser tests with nothing skipped. The guarantees the commits claim were checked by breaking the guarded line and confirming a test failed: a prompt edit without a version bump, the interim review cap and its limit of 72, 401 and 403 provider refusal, the published-name check, judge case coverage, the C ownership note, and the version parity check. The candidate case fixes were reproduced before and after in Chromium. One upstream test this branch does not touch, setup_page_accepts_credentials_and_writes_the_primary_config_file in tests/cli.rs, failed once under load and passed on every rerun.

Deliberately left for follow-up: a report for an original exercise can still mention LeetCode, because the published-name check keys off a source title those exercises do not have; a sixth candidate case blocks the test run instead of being refused on its own; a null expected value breaks the palindrome checker; a mistyped class-case method breaks the whole C++ or Java compile; only four of the five candidate cases reach the reviewer; the parity check passes when neither version can be read; and the failed-review test cannot fail as written. A checkout that still holds web/vendor/avatar/jim.vrm from before the model moved to the browser cache now fails verify-vendor.sh until the file is deleted.

Summary by cubic

Expands practice feedback and report history while tightening problem-bank validation and release checks. The active interview contract is now bundle 11 with live prompt 3, report prompt 10, rubric 1, and report schema 2; hints are qualitative context for independence rather than a numeric communication-score deduction.

Practice and reports

Candidate-authored cases are type-checked, run separately from judge totals, capped at five, and reach the interviewer and report with their inputs, so each case can be identified.
Reports include debrief guidance, hint rung, volunteered hints, practiced level, topics, and neutral camera-skip conditions; published problem names are rejected or repaired.
Prompt-only edits no longer invalidate stored scores; rubric and schema versions determine compatibility, with golden coverage for prompt version bumps.
Full reports stay reopenable beyond the short history, due reviews survive level changes, failed reviews reset to one day, and progress groups by topic.
The interview page no longer evaluates strings: the runner uses a blob worker, and face detection moves to its own script, so the page policy grants eval to one worker instead of every script; storage is resolved through one accessor that answers null, so the page still loads when site data is blocked.
Interim report-model reviews are capped by CODETRIAL_MAX_INTERIM_REVIEWS (default 12, maximum 72); final reports retain complete context.

Problem bank and tooling

Original exercises are supported, including generated rubrics and the new Bounded Event Queue problem; reference notes are renamed with the scenario, and imported exercises keep published-source metadata out of candidate-facing data.
Judges must provide five executable cases with a type-valid boundary case; C array-return starters must document caller ownership; class harnesses validate the starter's class name and arity and derive return types from contracts.
Providers with fresh 401 or 403 responses leave rotation until credentials change and the server restarts.
CI checks Rust release-image parity and scheduled avatar model delivery; hook and vendor migration paths are removed, and stale web/vendor/avatar/jim.vrm files must be deleted.
CLI tests bound concurrent server processes and treat signal-killed processes as failures; the agent and web test suites split into per-area module files so a failing test is easier to place.

^{Written for commit 5d88528. Summary will update on new commits.}

A prompt edit used to invalidate stored scores, so every change had to be added to a browser-side list that nothing forced anyone to update. Compare the recorded rubric and schema versions instead: a prompt-only bump leaves scores comparable and the list goes away. The golden digest now covers every prompt builder, so an edit that skips its version bump fails the gate rather than a review.

A report could name the published exercise or the practice site it came from, which hands back the source the interview scenario deliberately withholds. Validate against the selected problem at the exact paths a candidate reads, and let Gemini repair a violation so one stray title does not cost the whole report.

A verdict arrived with nothing to learn from it. Carry debrief metadata on generated and fallback reports alike, under a second report schema so the addition is recorded rather than inferred. Schema 1 reports stay scored, because existing history is still the candidate's.

Spaced-repetition reviews were filtered by the level chosen for new problems, so changing level silently hid work already due. Choose due reviews before that filter applies. A failed review drops to a one-day interval, which is what being due is for, and outcomes are ordered by time first so an older failure cannot erase the successes recorded after it.

The full history keeps twenty attempts, so an older one disappeared from the lobby altogether rather than ageing out of detail. Keep a second list of full reports past that cap, bounded by a local-storage byte budget rather than a row count, and draw the lobby from both, paired on the id every saved report carries so an attempt held in both is listed once. A row predating that id is given one in each store the first time it is read. The short history is written first and alone: the report a candidate is waiting on must not lose its room to the larger store behind it, and a write refused there costs a later reopen rather than the save.

A candidate could only run the judge's cases, so the cases they thought to try left no trace in the evidence. Saved typed cases run beside the judge's without touching its totals, and up to five reach both the interviewer and the report prompt, where testing instinct is worth reading. A typed case is checked against the operations and arities the exercise declares, so a malformed one is refused with a reason instead of failing inside the runner, and a sixth is refused on its own rather than stopping the run it was typed into.

Every bank entry had to be an imported exercise carrying a published title and examples, which tied the bank to one study plan and left no way to write a problem for this tree. An entry may now declare itself original and carry no published fields: its title comes from the variant, validation stops demanding a source, and the plan checks pass it over. Rubrics become generated output, because the title an entry shows is no longer a constant the bank can state.

A hint count alone cannot separate a candidate who asked once for a deep hint from one the interviewer kept prompting unasked. Pass the rung reached and how many hints were volunteered, and say in the prompt that both are context: a volunteered hint is weaker evidence than a requested one and neither is a numeric deduction.

A missing or declined camera turned a candidate away from an interview that never needed video. Require one only when recording is enabled, since that is what the recording notice describes. The reason the camera went unused is signed into the integrity trail so the report reads a neutral condition rather than absence of evidence.

Quiet-pause reviews and the final report spend the same provider quota, and the existing rate limits still allow seventy-two reviews in a long interview, ahead of the report that matters. Bound them per interview, fixed when the interview starts so a config reload cannot raise the ceiling underneath a session already running.

The pool probe removed a project only for an explicit 429, so a project whose credential had been revoked stayed in rotation and failed later at token minting, where the candidate sees it. Read 401 and 403 as refusals too. Everything else still counts as available, because refusing an interview over a brief network fault is the worse error.

A candidate practising for a senior role got the same verdict as one practising for a junior role, with nothing on the page saying which bar had been applied. Pass the selected level as coaching context and show it beside the verdict. The decision stays against the fixed mid-level bar, which the prompt now states twice because it is what a reviewer is most likely to let drift.

Progress listed recurring weaknesses but never which topics had actually been practised, so a topic failed twice and a topic never attempted looked alike. Group attempts by the topics their problems carry, with passes and the date each was last seen.

A judge could ship with two cases and none of them near a boundary, which makes a passing run weak evidence of anything. Require five cases and one the declared argument types admit as small at their own domain, rather than treating a single JSON shape as a boundary everywhere. The exception file carries its own count, so a line left behind after its cases are authored fails instead of excusing that judge forever. The restored index contract now admits an empty tree, because a boundary case has to be one the exercise actually accepts.

A C starter that returns an array through returnSize says nothing about who frees it, and a candidate reading it has no way to find out. Require the starter comment to state that the caller frees the returned storage.

The release workflow pinned a toolchain that nothing compared against what the crate claims to build with, so the two could drift until a release failed. Declare the minimum in the manifest and fail the gate when the Linux image no longer matches it.

The avatar browser check accepts an unavailable avatar, so that a developer without the model can still run the flow. That also means nothing would notice the pinned model going away. Run the flow on a schedule with the fallback disallowed, which checks the pin somewhere without making the local run depend on it.

The hook installer and the vendor fetch carried branches for trees laid out the way this one no longer is, and nothing left in the repository can reach them. Require the current behavior outright. The docs go with them: the Java lane needs a working toolchain rather than a version nobody here tested, and the connection notes drop a claim about which assertions moved that was never checked.

Most tests here start a real server and the harness runs one thread per core, so a small machine was asked to hold more of them than it had memory for and the kernel picked them off. Take a slot before spawning and give it back when the process is reaped, charged to the thread rather than to each server: a test that stops one and starts another holds two, and charging both would let such tests take every slot and then wait on each other. Then stop reading a killed process as a clean exit. An exit code is absent when a signal ends a process, and both runners turned that absence into zero, so a usage error the binary refuses and a production start it declines both read as accepted.

Bundle 10 retires the instruction that every hint should meaningfully reduce the communication score. A hint is evidence about independence, but how much to subtract was left to the model, so two interviews with the same hint count were marked differently for it. Hints stay as context the reviewer weighs against everything else. The register in docs is now read back by the bundle test, because it is what a maintainer consults before a bump and it had nothing holding it to the constants.

The harness instantiates the judge's class name and the candidate defines it in their starter, but the two are produced from different files: the judge is renamed out of the bank, the starters are rewritten from the variant. A rename that reached one and not the other compiled to a missing symbol on the candidate's first run, and nothing compared them.

The interviewer and the report reviewer saw a candidate case by label and output alone, so "Your case 3: got [0, 1]" named a case neither of them could identify and neither could say whether the output was right for it. Bundle 11 carries the input beside the result. It is candidate-authored text, so it is truncated and stripped by the same reader as the output it sits next to. A run recorded before this carries no input, and says nothing rather than claiming one.

tests/agent.rs and tests/web.rs had grown past six and nine thousand lines, which made a failing test hard to place and every change to them a merge risk. Each stays the test target cargo builds, and its tests move into modules beside it grouped by what they exercise. Nothing is added or removed; fixture paths move one directory up with the files that include them.

The page policy kept 'unsafe-eval' because the JavaScript runner evaluated the candidate's code from a string inside a blob worker, and a blob worker inherits the document's policy. That granted eval to every script on the one page that renders the candidate's own text. The runner now builds the worker with the code as its first statements, which needs only blob:, and the page keeps 'wasm-unsafe-eval' for Pyodide alone. MediaPipe's glue still builds functions from text, so face detection runs in face-worker.js, the one response served a policy that allows it; the setup check routes there too instead of loading the bundle into the page, where it would fail and read as no detector.

optimal and pitfalls were written against the published problem and stayed unrenamed while every other piece of prose took the scenario's names. They are not reviewer-only: the live prompt interpolates both and the debrief stamps them into the report the candidate reads, so a published parameter name reached the candidate through either. They are now posed like the rest of the prose, and the source checks read them. Only parameters and terms are substituted, in one pass, since an entry point is usually an English verb. Two bank entries whose new names are adjectives are reworded so the result still reads.

Where site data is blocked, touching the sessionStorage global throws rather than returning a storage that refuses. The interview page read it at module scope, so the whole page failed to load, and the storage helpers named their default as a parameter, which is evaluated before their try. Storage is now resolved through one accessor that answers null there, and every reader already treats that as nothing stored.

The server-level test sent a token request once the stub had counted a probe and asserted no second one. The stub counts before the refresher writes its verdict, so a request landing in between probed inline and the test failed on timing alone. The server test now asserts only what it can observe, a probe with nobody asking, and the cache read moves to the unit test, where refresh_all is awaited and the probes are counted.