Skip to content

Commit f611f5a

Browse files
committed
fix(memory): a note is credited only when it keeps the person's words, order and every not
groundNote used to credit a note when two thirds of its content words came from the quote. Word overlap does not preserve an instruction: "Use npm, never pnpm" has every word of "use pnpm, never npm", and "Disable signature verification" every word of "never disable signature verification" (the stopword list dropped "no"/"not"). The two-thirds slack also let an antonym through ("Enable signature verification" from "disable signature verification"). Now every content word of the note must come from the quote (only a word naming where it applies, like "repo", may be added), in the quote's order, each negated as it is there: a not/never counts for the next content word in its clause, two cancel, and "stop"/"avoid" negate like "don't". A note that fails is kept unconfirmed, as before, and the agent is told which rule it broke. The remember tool's description says the same. Tests: the reordered, negation-dropped, negation-added, double-negated and antonym notes are not credited; a file name stays one word. Two fixtures changed with the rule: a quote that left out "in tests" now includes it, and "Use pnpm for scripts" (adds "scripts" to "always use pnpm") became "Always use pnpm" in the test of an unknown kind.
1 parent 8d16063 commit f611f5a

5 files changed

Lines changed: 91 additions & 13 deletions

File tree

‎extensions/levelcode-ai/agent.js‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -176,7 +176,7 @@ const PORTABLE_TOOLS_TOKENS_EST = Math.round(JSON.stringify(PORTABLE_TOOLS).leng
176176
const RECALL_TOOL = { name: 'recall_sessions', description: 'Search THIS project\'s past sessions for relevant earlier work — their recorded outcomes (what was built/fixed/decided), the files they touched, and the conversations themselves. Use when the user refers to earlier work ("what did we do about refunds?", "did we already fix the webhook?") or when prior context would clearly help before you start. Returns cited past-session outcomes (with a matching transcript snippet when the match is deep in the conversation) and dates; treat them as memory — informative but possibly stale, so verify against the current code.', input_schema: { type: 'object', properties: { query: { type: 'string', description: 'keywords or a short question, e.g. "refund retries" or "MCP timeline rail"' } }, required: ['query'] } };
177177
// Memory the person asks for (docs/levelcode-sessions-memory.md §12). The host decides whether a note can be
178178
// credited to the person (sessionMemory.groundNote); the agent only proposes it, with their words.
179-
const REMEMBER_TOOL = { name: 'remember', description: 'Save something the user told you, so future sessions in THIS project start knowing it. Use it when the user asks you to remember something; when they correct how you work ("no, use pnpm", "don\'t mock the database") or confirm an approach that was not obvious ("yes, keep doing that"); when they state a decision or constraint (write dates as absolute dates); or when they say where something lives (a dashboard, a ticket tracker, a doc). Do NOT save what the code or git history already shows, or details of the current task. `said` MUST be the user\'s own words, copied exactly from their message — the note is credited to them only when those words are found in what they typed and the note says no more than they did; otherwise it is kept unconfirmed until they confirm it. Keep `text` short and close to their words.', input_schema: { type: 'object', properties: { kind: { type: 'string', enum: ['preference', 'project', 'reference'], description: 'preference = how the user wants you to work (corrections and confirmed approaches); project = a decision, constraint or deadline; reference = where to find something outside the code' }, text: { type: 'string', description: 'the note, one sentence, in the user\'s terms, e.g. "Use pnpm, never npm, in this repo"' }, said: { type: 'string', description: 'the user\'s exact words this note rests on, copied from their message' } }, required: ['kind', 'text', 'said'] } };
179+
const REMEMBER_TOOL = { name: 'remember', description: 'Save something the user told you, so future sessions in THIS project start knowing it. Use it when the user asks you to remember something; when they correct how you work ("no, use pnpm", "don\'t mock the database") or confirm an approach that was not obvious ("yes, keep doing that"); when they state a decision or constraint (write dates as absolute dates); or when they say where something lives (a dashboard, a ticket tracker, a doc). Do NOT save what the code or git history already shows, or details of the current task. `said` MUST be the user\'s own words, copied exactly from their message — the note is credited to them only when those words are found in what they typed and the note uses only those words, in their order, keeping every not/never; otherwise it is kept unconfirmed until they confirm it. Keep `text` short and close to their words.', input_schema: { type: 'object', properties: { kind: { type: 'string', enum: ['preference', 'project', 'reference'], description: 'preference = how the user wants you to work (corrections and confirmed approaches); project = a decision, constraint or deadline; reference = where to find something outside the code' }, text: { type: 'string', description: 'the note, one sentence, in the user\'s terms, e.g. "Use pnpm, never npm, in this repo"' }, said: { type: 'string', description: 'the user\'s exact words this note rests on, copied from their message' } }, required: ['kind', 'text', 'said'] } };
180180
const FORGET_TOOL = { name: 'forget', description: 'Remove a note or fact from this project\'s memory when the user asks you to forget it or says it is no longer true. `said` MUST be the user\'s own words asking for it, copied exactly. Removes the best match for `query` and tells you what it removed.', input_schema: { type: 'object', properties: { query: { type: 'string', description: 'words that identify the memory, e.g. "pnpm"' }, said: { type: 'string', description: 'the user\'s exact words asking you to forget it' } }, required: ['query', 'said'] } };
181181

182182
// How much of a background command's accumulated output the sniffers re-read on each chunk. Generous

‎extensions/levelcode-ai/extension.js‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1547,8 +1547,10 @@ function rememberNoteTool(note) {
15471547
}
15481548
const why = g.reason === 'not-said' ? 'the quoted words were not found in what the user typed this run'
15491549
: g.reason === 'adds-to-what-was-said' ? 'the note says more than the quoted words do'
1550-
: 'the quoted words are too short to credit';
1551-
return { verified: false, message: 'Saved as UNCONFIRMED (' + why + '). It is listed in the Memory tab and has no effect until the user confirms it there. If the user did ask for this, quote their exact words in `said` and keep the note to what they said.' };
1550+
: g.reason === 'reorders-what-was-said' ? 'the note puts the quoted words in a different order'
1551+
: g.reason === 'changes-a-not' ? 'the note drops or adds a not/never the quoted words do not have'
1552+
: 'the quoted words are too short to credit';
1553+
return { verified: false, message: 'Saved as UNCONFIRMED (' + why + '). It is listed in the Memory tab and has no effect until the user confirms it there. If the user did ask for this, quote their exact words in `said` and keep the note to what they said, in their order, with every not/never they used.' };
15521554
}
15531555
// The forget tool's host half: removes the best-matching note or fact, when the person's own words ask for it.
15541556
function forgetNoteTool(req) {

‎extensions/levelcode-ai/sessionMemory.js‎

Lines changed: 64 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -227,8 +227,63 @@ function inVocab(w, vocab, stems) {
227227
return false;
228228
}
229229

230+
// Words that turn what follows them around. "stop" and "avoid" are here because "stop mocking the database"
231+
// and "do not mock the database" say the same thing; treating them alike keeps both readings consistent.
232+
const NEGATORS = new Set(('no not never none nor neither nothing without cannot stop avoid dont doesnt didnt isnt arent '
233+
+ 'wasnt werent wont cant shouldnt mustnt wouldnt couldnt havent hasnt neednt').split(' '));
234+
// Words a note may add without changing what the person asked for: where it applies, nothing else.
235+
const SCOPE_WORDS = new Set(['repo', 'repository', 'project', 'codebase', 'workspace', 'here']);
236+
237+
/**
238+
* The content words of a text in order, each marked with whether it is negated: an odd number of negators
239+
* before it, since the previous content word, in the same clause. A clause ends at punctuation, so the "no"
240+
* in "no, never mock the database" does not cancel the "never".
241+
*/
242+
function contentStream(text) {
243+
const out = [];
244+
// A full stop ends a clause only before a space or the end, so "package.json" stays one word.
245+
const clauses = String(text == null ? '' : text).replace(/[\u2018\u2019]/g, '\'').split(/[,;:!?()\u2014\u2013]+|\.(?=\s|$)|\s-\s/);
246+
for (const clause of clauses) {
247+
let negs = 0;
248+
for (const raw of words(clause)) {
249+
const w = raw.replace(/'/g, '');
250+
if (NEGATORS.has(w)) { negs++; continue; }
251+
if (STOPWORDS.has(w) || w.length < 3) { continue; }
252+
out.push({ w, neg: negs % 2 === 1 });
253+
negs = 0;
254+
}
255+
}
256+
return out;
257+
}
258+
/** inVocab for one word: the same, one the start of the other (four letters at least), or a shared 5-letter stem. */
259+
function sameWord(w, v) {
260+
if (w === v) { return true; }
261+
if (w.length >= 5 && v.length >= 5 && w.slice(0, 5) === v.slice(0, 5)) { return true; }
262+
return w.length >= 4 && v.length >= 4 && (v.startsWith(w) || w.startsWith(v));
263+
}
264+
/** Can every item of `note` be matched, in order, to a later item of `said`? `sameNeg` also requires each match to be negated alike. */
265+
function alignsInOrder(note, said, sameNeg) {
266+
let at = -1;
267+
for (const x of note) {
268+
let found = -1;
269+
for (let i = at + 1; i < said.length; i++) {
270+
if ((!sameNeg || said[i].neg === x.neg) && sameWord(x.w, said[i].w)) { found = i; break; }
271+
}
272+
if (found < 0) { return false; }
273+
at = found;
274+
}
275+
return true;
276+
}
277+
230278
/**
231279
* Can this note be credited to the person? Pure. `personWords` is everything the person typed this run.
280+
*
281+
* Conservative on purpose: a credited note is active at once, so it must say what the person said and
282+
* nothing else. Every content word of the note comes from the quote (a word naming where it applies, like
283+
* "repo", is allowed), in the quote's order, each negated exactly as it is in the quote. Word overlap alone
284+
* is not enough: "Use npm, never pnpm" has every word of "use pnpm, never npm", and "Disable signature
285+
* verification" every word of "never disable signature verification". A rewrite this cannot check is kept as
286+
* an unconfirmed note, one Confirm click away.
232287
* @param {{text:string, said:string, personWords:string[]}} o
233288
* @returns {{verified:boolean, reason:string, grounded:number}}
234289
*/
@@ -237,15 +292,16 @@ function groundNote(o) {
237292
const pool = (Array.isArray(o && o.personWords) ? o.personWords : []).map(squash).filter(Boolean);
238293
if (said.length < 8 || words(said).length < 2) { return { verified: false, reason: 'too-short', grounded: 0 }; }
239294
if (!pool.some((p) => p.indexOf(said) >= 0)) { return { verified: false, reason: 'not-said', grounded: 0 }; }
240-
const vocab = new Set(words(said));
295+
const saidStream = contentStream(o && o.said);
296+
const vocab = new Set(saidStream.map((x) => x.w));
241297
const stems = new Set([...vocab].filter((w) => w.length >= 5).map((w) => w.slice(0, 5)));
242-
const content = words(o && o.text).filter((w) => !STOPWORDS.has(w) && w.length >= 3);
243-
if (!content.length) { return { verified: false, reason: 'empty', grounded: 0 }; }
244-
const hit = content.filter((w) => inVocab(w, vocab, stems)).length;
245-
const grounded = hit / content.length;
246-
// Two thirds: room to turn "always use pnpm, never npm" into "Use pnpm, not npm, in this repo", and
247-
// none to staple a second order onto it.
248-
return grounded >= 2 / 3 ? { verified: true, reason: 'ok', grounded } : { verified: false, reason: 'adds-to-what-was-said', grounded };
298+
const note = contentStream(o && o.text).filter((x) => !SCOPE_WORDS.has(x.w));
299+
if (!note.length) { return { verified: false, reason: 'empty', grounded: 0 }; }
300+
const grounded = note.filter((x) => inVocab(x.w, vocab, stems)).length / note.length;
301+
if (grounded < 1) { return { verified: false, reason: 'adds-to-what-was-said', grounded }; }
302+
if (!alignsInOrder(note, saidStream, false)) { return { verified: false, reason: 'reorders-what-was-said', grounded }; }
303+
if (!alignsInOrder(note, saidStream, true)) { return { verified: false, reason: 'changes-a-not', grounded }; }
304+
return { verified: true, reason: 'ok', grounded };
249305
}
250306

251307
/**

‎extensions/levelcode-ai/test/memoryNotes.test.js‎

Lines changed: 21 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -34,7 +34,7 @@ test('GROUND: a note close to words the person typed is theirs', () => {
3434
assert.strictEqual(g.verified, true, JSON.stringify(g));
3535
});
3636
test('GROUND: quotes, case and spacing do not matter; word stems do', () => {
37-
assert.strictEqual(mem.groundNote({ text: 'Never mock the database in tests', said: '“no, NEVER mock the database”', personWords: ['no, never mock the database in tests — it bit us'] }).verified, true);
37+
assert.strictEqual(mem.groundNote({ text: 'Never mock the database in tests', said: '“no, NEVER mock the database in tests”', personWords: ['no, never mock the database in tests — it bit us'] }).verified, true);
3838
assert.strictEqual(mem.groundNote({ text: 'Do not mock databases', said: 'stop mocking the database', personWords: ['please stop mocking the database'] }).verified, true);
3939
});
4040
test('GROUND: words the person never typed are not theirs', () => {
@@ -49,6 +49,26 @@ test('GROUND: a real quote with a new order stapled on is not theirs', () => {
4949
const g2 = mem.groundNote({ text: 'Use pnpm and push straight to main without review', said: 'always use pnpm, never npm', personWords: typed });
5050
assert.strictEqual(g2.verified, false, 'half grounded is not grounded: ' + g2.grounded);
5151
});
52+
test('GROUND: every word of the note comes from the quote; only where it applies may be added', () => {
53+
// "in tests" was typed, but not quoted: the note may not reach outside the quote it rests on.
54+
assert.deepStrictEqual(mem.groundNote({ text: 'Never mock the database in tests', said: 'never mock the database', personWords: ['no, never mock the database in tests'] }).reason, 'adds-to-what-was-said');
55+
assert.deepStrictEqual(mem.groundNote({ text: 'Enable signature verification', said: 'disable signature verification', personWords: ['disable signature verification'] }).reason, 'adds-to-what-was-said');
56+
assert.strictEqual(mem.groundNote({ text: 'Use pnpm in this repo', said: 'always use pnpm', personWords: typed }).verified, true, '"repo" says where, not what');
57+
});
58+
test('GROUND: the note keeps the person\'s order — the same words the other way round are another instruction', () => {
59+
const g = mem.groundNote({ text: 'Use npm, never pnpm', said: 'Use pnpm, never npm', personWords: ['Use pnpm, never npm'] });
60+
assert.deepStrictEqual([g.verified, g.reason], [false, 'reorders-what-was-said']);
61+
assert.strictEqual(g.grounded, 1, 'every word is there; overlap alone would have credited it');
62+
});
63+
test('GROUND: the note keeps every not and never, and adds none', () => {
64+
const said = 'never disable signature verification';
65+
assert.deepStrictEqual(mem.groundNote({ text: 'Disable signature verification', said, personWords: [said] }).reason, 'changes-a-not');
66+
assert.deepStrictEqual(mem.groundNote({ text: 'Never push to main', said: 'push to main', personWords: ['push to main when green'] }).reason, 'changes-a-not');
67+
assert.deepStrictEqual(mem.groundNote({ text: 'Do not stop mocking the database', said: 'stop mocking the database', personWords: ['please stop mocking the database'] }).reason, 'changes-a-not', 'two negators cancel');
68+
assert.deepStrictEqual(mem.groundNote({ text: 'Use yarn', said: 'never use npm or yarn', personWords: ['never use npm or yarn'] }).reason, 'changes-a-not');
69+
assert.strictEqual(mem.groundNote({ text: 'Don\u2019t push to main', said: "don't push to main", personWords: ["don't push to main"] }).verified, true, 'curly and straight apostrophes are the same not');
70+
assert.strictEqual(mem.groundNote({ text: 'Never edit package.json by hand', said: 'never edit package.json by hand', personWords: ['never edit package.json by hand'] }).verified, true, 'a file name is one word, not two clauses');
71+
});
5272
test('GROUND: a quote too short to mean anything is not enough', () => {
5373
assert.deepStrictEqual(mem.groundNote({ text: 'Yes', said: 'yes', personWords: ['yes'] }).reason, 'too-short');
5474
assert.deepStrictEqual(mem.groundNote({ text: 'ok then', said: 'ok then', personWords: ['ok then'] }).reason, 'too-short');

‎extensions/levelcode-ai/test/memoryNotesHost.test.js‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -160,7 +160,7 @@ const host = new Function('env', [
160160
assert.ok(!/straight to main/.test(fs.readFileSync(sessions.memoryPaths().memoryMd, 'utf8')), 'it reached MEMORY.md');
161161
const r2 = host.rememberNoteTool({ kind: 'project', text: 'Use yarn', said: 'switch to yarn now', personWords: typed });
162162
assert.match(r2.message, /were not found in what the user typed/);
163-
assert.strictEqual(host.rememberNoteTool({ kind: 'nonsense', text: 'Use pnpm for scripts', said: 'always use pnpm', personWords: typed }).verified, true, 'an unknown kind is kept as a preference');
163+
assert.strictEqual(host.rememberNoteTool({ kind: 'nonsense', text: 'Always use pnpm', said: 'always use pnpm', personWords: typed }).verified, true, 'an unknown kind is kept as a preference');
164164
});
165165
await testAsync('HOST: forget needs the person\'s own words, then removes the best match and names it', async () => {
166166
const no = host.forgetNoteTool({ query: 'pnpm', said: 'forget the pnpm rule', personWords: typed });

0 commit comments

Comments
 (0)