-
-
Notifications
You must be signed in to change notification settings - Fork 561
fix(modes): ticking "Reference files" in Technical Interview did not actually ground in them #521
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
143 changes: 143 additions & 0 deletions
143
electron/services/__tests__/InterviewPrepReferenceSwitch2026_08_29.test.mjs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,143 @@ | ||
| // Ticking "Reference files" in an interview-prep mode now actually grounds in | ||
| // them. | ||
| // | ||
| // THE GAP. T8 (2026-08-28) gave technical-interview a reference pool and put | ||
| // `reference_files` into its permitted switches, so the "Primary knowledge | ||
| // source" control offers it and an attached file became REACHABLE — measured | ||
| // second-person reachability went 0/6 to 6/6. But `buildUserSourceContract` | ||
| // pinned `defaultOwner: 'profile'` for interview-prep templates regardless of | ||
| // what the user ticked, so the saved contract still resolved `profile_only`, | ||
| // `documentGroundedFromContract` still returned false, and | ||
| // `forceDocumentGrounding` stayed OFF. | ||
| // | ||
| // Measured before this fix — the identical user selection, two modes: | ||
| // | ||
| // technical-interview + reference_files -> profile_only docGrounded=false | ||
| // general + reference_files -> reference_files_primary docGrounded=true | ||
| // | ||
| // So everything gated on that switch stayed off in the one mode whose users are | ||
| // most likely to upload project documents: topK 6 and a 1800-token budget | ||
| // instead of 12/3600, no per-file floor, no answerability scoring, no | ||
| // section-target or positional restore, no identity block, no query | ||
| // normalization. The user could ask for their reference files and be handed a | ||
| // materially weaker retrieval than the same files in General. | ||
| // | ||
| // UPLOAD IS STILL NOT CONSENT. This reads the user's EXPLICIT switch, never the | ||
| // presence of a file — the rule T8 was built around is unchanged and is | ||
| // asserted below. | ||
|
|
||
| import { test, describe } from 'node:test'; | ||
| import assert from 'node:assert/strict'; | ||
| import path from 'node:path'; | ||
| import { createRequire } from 'node:module'; | ||
| import { fileURLToPath } from 'node:url'; | ||
|
|
||
| const __dirname = path.dirname(fileURLToPath(import.meta.url)); | ||
| const repoRoot = path.resolve(__dirname, '../../..'); | ||
| const cjsRequire = createRequire(import.meta.url); | ||
|
|
||
| const { ModesManager } = cjsRequire(path.resolve(repoRoot, 'dist-electron/electron/services/ModesManager.js')); | ||
| const msc = cjsRequire(path.resolve(repoRoot, 'dist-electron/electron/services/modeSourceContract.js')); | ||
|
|
||
| const ENV = 'NATIVELY_RETRIEVAL_INTERVIEW_PREP_HONORS_REFERENCE_SWITCH'; | ||
| const withFlag = (value, fn) => { | ||
| const original = process.env[ENV]; | ||
| process.env[ENV] = value; | ||
| try { return fn(); } finally { | ||
| if (original === undefined) delete process.env[ENV]; | ||
| else process.env[ENV] = original; | ||
| } | ||
| }; | ||
|
|
||
| /** The real builder, without standing up a database. */ | ||
| const build = (templateType, switches) => | ||
| ModesManager.prototype.buildUserSourceContract.call( | ||
| Object.create(ModesManager.prototype), | ||
| { modeId: 'm', templateType, switches }); | ||
|
|
||
| const INTERVIEW_PREP = ['technical-interview', 'looking-for-work']; | ||
|
|
||
| describe('an explicit reference-files switch grounds interview-prep modes', () => { | ||
| for (const template of INTERVIEW_PREP) { | ||
| test(`${template}: ticking reference files enables document grounding`, () => { | ||
| const c = build(template, ['profile', 'job_description', 'reference_files']); | ||
| assert.equal(c.sourceAuthority, 'reference_files_primary'); | ||
| assert.equal(msc.documentGroundedFromContract(c, true), true, | ||
| 'forceDocumentGrounding must now be reachable for this mode'); | ||
| }); | ||
|
|
||
| test(`${template}: reference files ALONE also grounds`, () => { | ||
| const c = build(template, ['reference_files']); | ||
| assert.equal(msc.documentGroundedFromContract(c, true), true); | ||
| }); | ||
| } | ||
|
|
||
| test('it matches what the same selection already did in General', () => { | ||
| // The asymmetry was the bug: identical user intent, opposite outcome. | ||
| const ti = build('technical-interview', ['reference_files']); | ||
| const general = build('general', ['reference_files']); | ||
| assert.equal(ti.sourceAuthority, general.sourceAuthority); | ||
| }); | ||
| }); | ||
|
|
||
| describe('NON-REGRESSION: upload is not consent, and profile-first is still the default', () => { | ||
| for (const template of INTERVIEW_PREP) { | ||
| test(`${template}: WITHOUT the switch it stays profile-first`, () => { | ||
| const c = build(template, ['profile', 'job_description']); | ||
| assert.equal(c.defaultOwner, 'profile'); | ||
| assert.equal(c.sourceAuthority, 'profile_only'); | ||
| assert.equal(msc.documentGroundedFromContract(c, true), false, | ||
| 'a mode the user has not switched must not document-ground'); | ||
| }); | ||
|
|
||
| test(`${template}: no switches at all stays profile-first`, () => { | ||
| assert.equal(build(template, []).sourceAuthority, 'profile_only'); | ||
| }); | ||
| } | ||
|
|
||
| test('non-interview templates are completely unaffected', () => { | ||
| for (const t of ['general', 'sales', 'recruiting', 'team-meet', 'lecture']) { | ||
| const on = build(t, ['reference_files']); | ||
| const off = withFlag('0', () => build(t, ['reference_files'])); | ||
| assert.deepEqual(on, off, `${t} changed`); | ||
| } | ||
| }); | ||
|
|
||
| test('the kill switch restores the pre-fix behaviour exactly', () => { | ||
| withFlag('0', () => { | ||
| for (const template of INTERVIEW_PREP) { | ||
| const c = build(template, ['profile', 'job_description', 'reference_files']); | ||
| assert.equal(c.sourceAuthority, 'profile_only'); | ||
| assert.equal(msc.documentGroundedFromContract(c, true), false); | ||
| } | ||
| }); | ||
| }); | ||
| }); | ||
|
|
||
| describe('the memory-policy consequence is real, and is pinned here on purpose', () => { | ||
| // Switching an interview-prep mode to reference-files ownership ALSO changes | ||
| // its memory policy, because invariant #3 forbids Hindsight for any | ||
| // document-grounded owner. That is a deliberate consequence of the user's own | ||
| // choice, not a side effect to discover later in a bug report: | ||
| // | ||
| // allowHindsight true -> FALSE (cross-meeting recall is off) | ||
| // allowPriorAssistantFacts true -> FALSE (the assistant's own prior output | ||
| // stops counting as evidence — a | ||
| // fabrication vector closing, and | ||
| // arguably a gain) | ||
| // allowPriorAssistantReferents stays TRUE, so follow-ups still resolve | ||
| // ("what did you monitor after that?") | ||
| test('grounding an interview-prep mode turns Hindsight off', () => { | ||
| const grounded = build('technical-interview', ['reference_files']); | ||
| assert.equal(grounded.memoryPolicy.allowHindsight, false); | ||
| assert.equal(grounded.memoryPolicy.allowPriorAssistantFacts, false); | ||
| assert.equal(grounded.memoryPolicy.allowPriorAssistantReferents, true, | ||
| 'follow-up resolution must survive — losing it would break "and what about X?"'); | ||
| }); | ||
|
|
||
| test('an ungrounded interview-prep mode keeps both', () => { | ||
| const profileFirst = build('technical-interview', ['profile']); | ||
| assert.equal(profileFirst.memoryPolicy.allowHindsight, true); | ||
| assert.equal(profileFirst.memoryPolicy.allowPriorAssistantFacts, true); | ||
| }); | ||
| }); |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
When an interview-prep mode is saved with
reference_filesalongsideprofileorjob_description, this changes the owner toreference_files; the resulting capability branch forbids résumé, project, and job-description evidence, causing answers to ignore sources the user explicitly selected.Knowledge Base Used: