docs(bugs): answering pipeline bug reports — Aug 2026 (Telegram user issues) - #429
Open
Fork-You-Later wants to merge 6 commits into
Open
Conversation
…ative similarity
Add two focused review documents covering findings from an analysis of the
answering pipeline in IntelligenceEngine.ts and the speculative pre-fetch
similarity gate:
- intelligence-engine-architecture-critique.md
* runWhatShouldISay is ~2,400 lines — extract into WtaPipeline stages
* Post-stream repair cascade: up to 3 sequential blocking LLM regen calls
after the answer already streamed to the UI (worst case +14s latency)
* Disabled answer relevance guard (Phase 8) still runs NLI on every answer
* AnswerPlanner.ts at 211KB — needs module split
* Manual/WTA post-processing duplicated between IntelligenceEngine and ipcHandlers
* Regex-based LLM failure detection growing per provider/version
* Dynamic require() in hot path with fail-open catch
* Inconsistent sanitization limits copy-pasted at 3 repair sites
- speculative-similarity-jaccard-critique.md
* Jaccard token-overlap inflated by stop words on interview questions
* Antonym pairs (strengths/weaknesses, success/failure) score ~0.75 — false accept
* Recommended fix: hybrid Jaccard fast-exit + SBERT (all-MiniLM-L6-v2, 22MB)
in the ambiguous zone, reusing IntentClassifier.ts worker infrastructure
* Implementation sketch and threshold calibration guidance included
No production code was changed. These are read-only findings for implementing devs.
- Correct Jaccard scores: succeeded/failed scores ~0.77 (false accept under 0.75 threshold), strengths/weaknesses scores ~0.72 (close to threshold) - Keep raw SBERT cosine scale consistent rather than normalizing it to [0,1] which would let a raw score of 0.31 pass the 0.65 threshold - Reference LocalEmbeddingProvider instead of IntentClassifier for retrieving SBERT sentence embeddings
Add replace-placeholder-corruption-bug.md detailing a critical issue where untrusted model code/math containing $ symbols (like bash $1, jQuery $&, or math $x_1) get corrupted during post-processing token restoration due to unsafe usage of JS String.prototype.replace() with string replacements. Update the index README and severity summary table accordingly.
Update speculativeSimilarity to return an explicit accepted boolean, resolving the contradiction where the caller compared the raw SBERT cosine score against the 0.75 Jaccard threshold (which would reject valid SBERT matches in the 0.65-0.75 range).
Three user-reported bugs documented with root cause analysis, proposed fixes, and jcodemunch blast-radius assessments: - Bug 001: documentGroundedCustomModeActive fires without hasReferenceFiles causing general-question refusals in doc-grounded modes - Bug 002: hasScreenContext not propagated from imagePaths in manual chat, breaking code generation from attached screenshots - Bug 003: V3 engine nullish-coalesces activeSkill away, silently dropping skill injection in v2.8.5+ documentGroundedFromContract() already has the correct guard but has zero call-sites -- recommend wiring it in for Bug 001.
|
| Filename | Overview |
|---|---|
| docs/bugs/bug-001-document-grounded-refusal.md | Documents the document-grounding failure and now safely guards the optional source contract in its preferred patch. |
| docs/bugs/bug-002-screenshot-code-generation.md | Documents missing manual-screenshot context propagation and proposes updates for both answering entry points. |
| docs/bugs/bug-003-skill-injection-failure.md | Documents how V3 prompt selection discards active skill instructions and proposes preserving both prompt components. |
Reviews (2): Last reviewed commit: "docs: address greptile bot review commen..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
…t#429 - bug-001: fix TypeScript undefined risk in preferred fix diff. input.activeMode?.sourceContract is ModeSourceContract | undefined; documentGroundedFromContract requires a concrete non-null contract. Added ternary null guard so call is only made when contract exists. - replace-placeholder-corruption-bug: correct factually wrong dollar-sign examples. With a literal string first arg and no capturing groups, ECMAScript specifies that dollar-n patterns (\, \) are left literal (NOT expanded). Only \$&, \, and \$' are real vulnerabilities. Removed incorrect Bash and math examples; replaced with accurate \$&, \, and \$' examples. Added explicit clarifying note. - speculative-similarity-jaccard-critique: label all SBERT threshold values (0.65, 0.92, cosine scores) as illustrative/uncalibrated. Added warning block after embedding examples, inline code comment, updated section header and summary table.
evinjohnn
pushed a commit
that referenced
this pull request
Aug 15, 2026
…pute docs Sweeps up what was left untracked in the working directory, with one deliberate exclusion. gitignore — the reason this is not a routine chore: KAUSHAL_12_DISPUTE_HANDOVER.md sits at the repo ROOT, a byte-identical stray copy of the file already held in the correctly-ignored dispute-evidence-kaushal-shivaprakashan/. The existing rules are all directory-scoped (/dispute-evidence-*), so the root copy was fully committable and the next `git add -A` would have published it. This repo is PUBLIC and that document carries a named customer's email address, phone number and home address. Adds /*_DISPUTE_HANDOVER.md and /*_DISPUTE_*.md so the root copies can never be staged. README.md and other root docs stay committable — the rule is scoped, not a blanket. Tests (new, previously untracked alongside 279 tracked siblings): V3SkillInjection2026_08_05 — PR #429 Bug 003, skill injection silently dropped when Context Intelligence V3 is active (default ON since 2026-07-30), covering both drop sites independently. ManualScreenshotScreenContext2026_08_05 — manual screenshot screen context. natively-api: cecbfdf → bedc55e, which is merged to that repo's main via PR #7 and is the deployed revision (Railway reports Online, /health 200). transitions/: 27 CSS transition reference docs. Scanned for credentials and customer data before publishing — the single hit is the word "password" inside a prose description of form-validation UX. These are tooling reference material rather than application source; say the word and they come back out. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SiDsWPD5XtBYgXWt7ZPizu
evinjohnn
pushed a commit
that referenced
this pull request
Aug 16, 2026
…Bug 003)
WhatToAnswerLLM composed its system prompt as
const _wtaSystemPrompt = _v3p?.system ?? finalPromptOverride;
finalPromptOverride is the ONLY carrier of the `## ACTIVE SKILL` block (it is
built as `${basePrompt}\n\n## ACTIVE SKILL\n${activeSkill.promptBlock}` at
WhatToAnswerLLM.ts:588). Context Intelligence V3 has been default ON since
2026-07-30, so _v3p is always defined, `??` never fell through, and the skill
block was dropped on every single turn — the user picked a skill, the UI showed
it active, and the model never saw it.
The two inputs are not interchangeable (V3's system prompt is a different
governed composition, not a superset), so composeWtaSystemPrompt APPENDS the
skill to whichever base actually rides the turn instead of picking one:
- no V3 prompt -> legacy override verbatim (it already holds the block)
- V3 prompt, no skill -> exactly the V3 prompt, byte-identical to before
- V3 prompt + skill -> V3 prompt + the ACTIVE SKILL block
Non-skill turns are therefore inert, which is what keeps this safe to land.
Kept as its own pure module so it is testable without constructing the whole
LLM graph, and reuses the same `## ACTIVE SKILL` heading as the legacy path.
The manual-chat half of Bug 003 (ipcHandlers: the /skill prefix is parsed ~35k
chars AFTER the V3 branch, so under V3 the prefix leaks to the model as literal
text) is NOT fixed here — separate commit.
Validation: typecheck clean under TS 7.0.2; the WTA half of
V3SkillInjection2026_08_05 now passes (5/8 -> the 3 remaining are the
manual-chat assertions).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UamHk73hUw1Fxevh8P2oDe
evinjohnn
pushed a commit
that referenced
this pull request
Aug 16, 2026
Symptom: 'screenshot attached but code not generated'. A manually-attached screenshot never set hasScreenContext, so the V3 turn classifier never added SCREEN_SPECIFIC / SCREEN_FACT and the screen was not treated as authoritative evidence — for exactly the turns where the screen is most obviously the subject. Two drop sites, both one-line: IntelligenceEngine.ts:2592 (WTA) — `Boolean(options?.screenContext)` reads the PERIODIC-CAPTURE OCR object. A hand-attached screenshot rides in imagePaths with screenContext null, so this was always false. Widened with the same predicate the legacy path already uses for _wtaHasVisualContext (~line 1228), so OCR turns are unaffected and only attached-image turns change. ipcHandlers.ts (manual chat) — hasScreenContext was omitted from the buildV3Prompt call entirely, defaulting to false. Manual chat has no periodic OCR object at all, so imagePaths is the only screen signal on that surface. Also documents V3SkillInjection's still-open manual-chat half in its header: both sites live in the same gemini-chat handler (V3 branch ~50356, skill parse ~85936), the block's dependencies are all available before the branch, and the known fallout is the legacy identity probe. Not auto-fixed because hoisting changes what every read of `message` between those offsets sees, and its `skillParseIdx < v3EntryIdx` assertion pins SOURCE OFFSETS — satisfiable by an occurrence that fixes nothing, which is the exact failure mode this branch spent its time repairing. Validation: typecheck clean under TS 7.0.2; both ManualScreenshotScreenContext tests pass. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UamHk73hUw1Fxevh8P2oDe
evinjohnn
pushed a commit
that referenced
this pull request
Aug 16, 2026
…refusals Two of the three remaining groups. ── Manual-chat skill injection (#429 Bug 003, second surface) ── The /skill-name parse lived ~35k characters BELOW the Context Intelligence V3 short-circuit in the same gemini-chat handler, so under V3 (default ON since 2026-07-30) the prefix reached the model as literal text and the skill instructions were injected nowhere. Hoisted the PARSE above the branch — but deliberately NOT the mutation. The hoisted block assigns to a new `skillStrippedMessage` and `message` is still mutated at the original boundary, so every reader in between (identity probe at ~80315, source-switch resolution, error logs) sees the user's literal input exactly as before. That is what makes a 3k-char relocation across a 12k-line hot path safe: the only consumer whose input changes is the V3 branch itself, which is the bug. Early skill errors now return before ForegroundGate.begin, which is null-guarded at release, and after stream registration exactly as before. The identity probe additionally gates on !skillPromptBlock. That immunity already existed but only incidentally — its anchored regexes could not match a "/humanize who are you" prefix — so it is now explicit and survives a regex relaxation. ── Doc-grounding refusal requires FILES ── The policy line keyed on enforcement alone, and enforcement is true on the AUTHORITY ALONE: strictDocumentGroundedFromContract returns true for `reference_files_only` before anything is uploaded. Such a mode was told to say the answer was "not in the uploaded material" when no uploaded material existed. Now requires enforcement AND files. I first implemented the stronger form this file originally asked for — suppression requires STRICT — and it broke F9PurposeHonestyUnderR1_2026_08_15. Investigating rather than reverting showed the two tests assert OPPOSITE output for an IDENTICAL plan (same answerType, enforcement, strict, files: verified directly), so no discriminator exists. R1 enforcement is what actually sets forceDocumentGrounding (LLMHelper.ts:2299, IntelligenceEngine.ts:1179), so retrieval IS forced for seeded+files and the honesty mandate is honest. The older assertions encoded a premise PR #466 superseded; they were rewritten with that reasoning recorded, not deleted quietly. Validation: typecheck clean under TS 7.0.2. V3SkillInjection 8/8 (was 5/8), DocGroundedRoutingRequiresFiles 11/11, F9PurposeHonestyUnderR1 3/3. Full-suite regression run from the hoist: zero new failures. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UamHk73hUw1Fxevh8P2oDe
Abitesh
pushed a commit
to Abitesh/Sneak-Peek
that referenced
this pull request
Aug 30, 2026
…pute docs Sweeps up what was left untracked in the working directory, with one deliberate exclusion. gitignore — the reason this is not a routine chore: KAUSHAL_12_DISPUTE_HANDOVER.md sits at the repo ROOT, a byte-identical stray copy of the file already held in the correctly-ignored dispute-evidence-kaushal-shivaprakashan/. The existing rules are all directory-scoped (/dispute-evidence-*), so the root copy was fully committable and the next `git add -A` would have published it. This repo is PUBLIC and that document carries a named customer's email address, phone number and home address. Adds /*_DISPUTE_HANDOVER.md and /*_DISPUTE_*.md so the root copies can never be staged. README.md and other root docs stay committable — the rule is scoped, not a blanket. Tests (new, previously untracked alongside 279 tracked siblings): V3SkillInjection2026_08_05 — PR Natively-AI-assistant#429 Bug 003, skill injection silently dropped when Context Intelligence V3 is active (default ON since 2026-07-30), covering both drop sites independently. ManualScreenshotScreenContext2026_08_05 — manual screenshot screen context. natively-api: cecbfdf → bedc55e, which is merged to that repo's main via PR Natively-AI-assistant#7 and is the deployed revision (Railway reports Online, /health 200). transitions/: 27 CSS transition reference docs. Scanned for credentials and customer data before publishing — the single hit is the word "password" inside a prose description of form-validation UX. These are tooling reference material rather than application source; say the word and they come back out. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SiDsWPD5XtBYgXWt7ZPizu
Abitesh
pushed a commit
to Abitesh/Sneak-Peek
that referenced
this pull request
Aug 30, 2026
…ely-AI-assistant#429 Bug 003) WhatToAnswerLLM composed its system prompt as const _wtaSystemPrompt = _v3p?.system ?? finalPromptOverride; finalPromptOverride is the ONLY carrier of the `## ACTIVE SKILL` block (it is built as `${basePrompt}\n\n## ACTIVE SKILL\n${activeSkill.promptBlock}` at WhatToAnswerLLM.ts:588). Context Intelligence V3 has been default ON since 2026-07-30, so _v3p is always defined, `??` never fell through, and the skill block was dropped on every single turn — the user picked a skill, the UI showed it active, and the model never saw it. The two inputs are not interchangeable (V3's system prompt is a different governed composition, not a superset), so composeWtaSystemPrompt APPENDS the skill to whichever base actually rides the turn instead of picking one: - no V3 prompt -> legacy override verbatim (it already holds the block) - V3 prompt, no skill -> exactly the V3 prompt, byte-identical to before - V3 prompt + skill -> V3 prompt + the ACTIVE SKILL block Non-skill turns are therefore inert, which is what keeps this safe to land. Kept as its own pure module so it is testable without constructing the whole LLM graph, and reuses the same `## ACTIVE SKILL` heading as the legacy path. The manual-chat half of Bug 003 (ipcHandlers: the /skill prefix is parsed ~35k chars AFTER the V3 branch, so under V3 the prefix leaks to the model as literal text) is NOT fixed here — separate commit. Validation: typecheck clean under TS 7.0.2; the WTA half of V3SkillInjection2026_08_05 now passes (5/8 -> the 3 remaining are the manual-chat assertions). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UamHk73hUw1Fxevh8P2oDe
Abitesh
pushed a commit
to Abitesh/Sneak-Peek
that referenced
this pull request
Aug 30, 2026
…I-assistant#429 Bug 002) Symptom: 'screenshot attached but code not generated'. A manually-attached screenshot never set hasScreenContext, so the V3 turn classifier never added SCREEN_SPECIFIC / SCREEN_FACT and the screen was not treated as authoritative evidence — for exactly the turns where the screen is most obviously the subject. Two drop sites, both one-line: IntelligenceEngine.ts:2592 (WTA) — `Boolean(options?.screenContext)` reads the PERIODIC-CAPTURE OCR object. A hand-attached screenshot rides in imagePaths with screenContext null, so this was always false. Widened with the same predicate the legacy path already uses for _wtaHasVisualContext (~line 1228), so OCR turns are unaffected and only attached-image turns change. ipcHandlers.ts (manual chat) — hasScreenContext was omitted from the buildV3Prompt call entirely, defaulting to false. Manual chat has no periodic OCR object at all, so imagePaths is the only screen signal on that surface. Also documents V3SkillInjection's still-open manual-chat half in its header: both sites live in the same gemini-chat handler (V3 branch ~50356, skill parse ~85936), the block's dependencies are all available before the branch, and the known fallout is the legacy identity probe. Not auto-fixed because hoisting changes what every read of `message` between those offsets sees, and its `skillParseIdx < v3EntryIdx` assertion pins SOURCE OFFSETS — satisfiable by an occurrence that fixes nothing, which is the exact failure mode this branch spent its time repairing. Validation: typecheck clean under TS 7.0.2; both ManualScreenshotScreenContext tests pass. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UamHk73hUw1Fxevh8P2oDe
Abitesh
pushed a commit
to Abitesh/Sneak-Peek
that referenced
this pull request
Aug 30, 2026
…refusals Two of the three remaining groups. ── Manual-chat skill injection (Natively-AI-assistant#429 Bug 003, second surface) ── The /skill-name parse lived ~35k characters BELOW the Context Intelligence V3 short-circuit in the same gemini-chat handler, so under V3 (default ON since 2026-07-30) the prefix reached the model as literal text and the skill instructions were injected nowhere. Hoisted the PARSE above the branch — but deliberately NOT the mutation. The hoisted block assigns to a new `skillStrippedMessage` and `message` is still mutated at the original boundary, so every reader in between (identity probe at ~80315, source-switch resolution, error logs) sees the user's literal input exactly as before. That is what makes a 3k-char relocation across a 12k-line hot path safe: the only consumer whose input changes is the V3 branch itself, which is the bug. Early skill errors now return before ForegroundGate.begin, which is null-guarded at release, and after stream registration exactly as before. The identity probe additionally gates on !skillPromptBlock. That immunity already existed but only incidentally — its anchored regexes could not match a "/humanize who are you" prefix — so it is now explicit and survives a regex relaxation. ── Doc-grounding refusal requires FILES ── The policy line keyed on enforcement alone, and enforcement is true on the AUTHORITY ALONE: strictDocumentGroundedFromContract returns true for `reference_files_only` before anything is uploaded. Such a mode was told to say the answer was "not in the uploaded material" when no uploaded material existed. Now requires enforcement AND files. I first implemented the stronger form this file originally asked for — suppression requires STRICT — and it broke F9PurposeHonestyUnderR1_2026_08_15. Investigating rather than reverting showed the two tests assert OPPOSITE output for an IDENTICAL plan (same answerType, enforcement, strict, files: verified directly), so no discriminator exists. R1 enforcement is what actually sets forceDocumentGrounding (LLMHelper.ts:2299, IntelligenceEngine.ts:1179), so retrieval IS forced for seeded+files and the honesty mandate is honest. The older assertions encoded a premise PR Natively-AI-assistant#466 superseded; they were rewritten with that reasoning recorded, not deleted quietly. Validation: typecheck clean under TS 7.0.2. V3SkillInjection 8/8 (was 5/8), DocGroundedRoutingRequiresFiles 11/11, F9PurposeHonestyUnderR1 3/3. Full-suite regression run from the hoist: zero new failures. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UamHk73hUw1Fxevh8P2oDe
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR documents three bugs in the answering pipeline surfaced by Telegram user reports in v2.8.5. No source-code changes — docs only, for dev review and targeted fix.
Bug 001 — Document-Grounded Refusal on General Questions 🔴
Root cause:
documentGroundedCustomModeActiveis read directly from the mode object without checkinghasReferenceFiles. Modes seeded withreference_files_primaryauthority fire the doc-grounded routing even when no files are uploaded, causing the WTA gate to fail against an emptydocContextBlockand return a refusal message.Key finding:
documentGroundedFromContract()inmodeSourceContract.tsalready implements the correctif (!hasReferenceFiles) return falseguard — it has zero call-sites.Fix surface:
AnswerPlanner.tsline 2335 (guard condition) +contextRoute.tsline 76 (secondary site).Bug 002 — Screenshot Attached but Code Not Generated 🟠
Root cause:
hasScreenContextis omitted frombuildV3Prompt()in thegemini-chat-streamIPC handler, andIntelligenceEngine.tsonly checksoptions.screenContext(the OCR object from the periodic capture service) — notimagePaths. Manually-attached screenshots never setoptions.screenContext, sohasScreenContextis alwaysfalsein manual chat.Fix surface: Single-line addition in
ipcHandlers.ts+ one-line guard expansion inIntelligenceEngine.ts.Bug 003 — Skill Injection Ignored in V3 Engine 🟠
Root cause:
WhatToAnswerLLM.tscomposes the system prompt as_v3p?.system ?? finalPromptOverride. When V3 is active,_v3pis always defined (non-nullish), sofinalPromptOverride— which carriesactiveSkill.promptBlock— is silently discarded. Skills appear to have no effect in v2.8.5.Fix surface: 3-line change in
WhatToAnswerLLM.ts— append skill block after V3 system prompt whenactiveSkillis present.Files in this PR
docs/bugs/README.mddocs/bugs/bug-001-document-grounded-refusal.mddocs/bugs/bug-002-screenshot-code-generation.mddocs/bugs/bug-003-skill-injection-failure.mdBlast Radius (jcodemunch)
documentGroundedCustomModeActivefans into 26 dependent files — all withhas_test_reach: false. Each bug doc includes a checklist of tests to add/update before merging the actual code fix.