fix(memory): word/sentence-boundary aware fact truncation - #12383
Open
LeMonBLOCK wants to merge 1 commit into
Open
fix(memory): word/sentence-boundary aware fact truncation#12383LeMonBLOCK wants to merge 1 commit into
LeMonBLOCK wants to merge 1 commit into
Conversation
sanitizeMatch() and capExtractionText() previously did raw character-offset slices (slice(0, MAX_FACT_LENGTH) / slice(-MAX_EXTRACTION_TEXT_LENGTH)) with no boundary awareness, producing garbled mid-word/mid-clause fragments that get injected into LLM context as memory facts. - sanitizeMatch() now backs the cut off to the nearest sentence-ending punctuation (. ! ?) within a lookback window, falling back to a plain whitespace boundary, falling back to the original hard cut only when no boundary exists nearby. - capExtractionText() applies the equivalent boundary-aware trim on the front edge of the kept tail. Mirrors the boundary-aware truncation pattern already used by open-sse/services/compression/lite.ts (diegosouzapw#8169) for tool-result truncation. Adds tests/unit/memory-extraction-boundary-truncation.test.ts covering word-boundary cuts, sentence-boundary preference, short-string passthrough, the no-boundary-available fallback, and capExtractionText's tail behavior.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bug
src/lib/memory/extraction.tstruncates extracted memory facts with rawcharacter-offset slicing and no boundary awareness:
sanitizeMatch()didraw.trim().replace(/\s+/g, ' ').slice(0, MAX_FACT_LENGTH)— a hard cut that can land mid-word.
capExtractionText()didtext.slice(-MAX_EXTRACTION_TEXT_LENGTH)on thefront edge when input exceeds
MAX_EXTRACTION_TEXT_LENGTH(64KB) — sameproblem, opposite edge.
These facts later get injected into LLM context as
<system-reminder>memory blocks, so a mid-word/mid-clause cut produces garbled fragments
(e.g. text starting with a stray closing parenthesis, or ending with no
punctuation and a half word).
Fix
sanitizeMatch()now backs the cut index off to the nearest cleanboundary within a lookback window: prefers a sentence-ending mark
(
./!/?) so the fact reads as a complete clause, falls back to aplain whitespace/word boundary, and only falls back to the original hard
cut when no boundary exists in the window (e.g. one long unbroken run of
characters).
capExtractionText()applies the equivalent boundary-aware trim on thefront edge of the kept tail (extends the cut forward to the next
whitespace boundary rather than slicing mid-word).
This mirrors the boundary-aware truncation already used by
open-sse/services/compression/lite.ts(issue #8169) for tool-resulttruncation, so the codebase now has one consistent pattern for this class
of problem.
Testing
tests/unit/memory-extraction-boundary-truncation.test.ts— coversa long match cut at a word boundary, a match cut preferentially at
sentence-ending punctuation, short strings passing through untouched,
the no-boundary-in-window fallback, and
capExtractionText'sboundary-aware tail behavior (both under and over the 64KB cap).
tests/unit/memory-extraction.test.tsalongside the newfile — all 28 tests pass, no regressions:
npx tsc --noEmit -p tsconfig.json— no new type errors introduced bythis change.
No unrelated files were touched.