-
-
Notifications
You must be signed in to change notification settings - Fork 559
docs(bugs): answering pipeline bug reports — Aug 2026 (Telegram user issues) #429
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
Fork-You-Later
wants to merge
6
commits into
Natively-AI-assistant:main
Choose a base branch
from
Fork-You-Later:docs/pipeline-bug-reports-aug2026
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from 5 commits
Commits
Show all changes
6 commits
Select commit
Hold shift + click to select a range
9f43e75
docs(tech-debt): architecture review of IntelligenceEngine and specul…
Fork-You-Later 107390e
docs(tech-debt): address PR review comments on Jaccard critique doc
Fork-You-Later 92c98ba
docs(tech-debt): document placeholder replacement corruption bug
Fork-You-Later e047856
docs(tech-debt): align speculative similarity thresholds in code sketch
Fork-You-Later abcc286
docs: add pipeline bug reports for Aug 2026 Telegram issues
Fork-You-Later 9b93c9a
docs: address greptile bot review comments on PR #429
Fork-You-Later File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,65 @@ | ||
| # Answering Pipeline Bug Report — August 2026 | ||
|
|
||
| ## Overview | ||
|
|
||
| This PR documents **three user-reported bugs** in the answering pipeline (v2.8.5). | ||
| All bugs were discovered from Telegram user reports and validated against the source code | ||
| with static analysis + jcodemunch blast-radius assessment. | ||
|
|
||
| **No source-code changes are in this PR** — these are documentation-only fixes for developer review. | ||
|
|
||
| --- | ||
|
|
||
| ## Bug Index | ||
|
|
||
| | ID | Title | Severity | Files to Fix | | ||
| |---|---|---|---| | ||
| | [Bug 001](./bug-001-document-grounded-refusal.md) | Document-Grounded Refusal on General Questions | 🔴 High | `AnswerPlanner.ts`, `contextRoute.ts` | | ||
| | [Bug 002](./bug-002-screenshot-code-generation.md) | Screenshot Attached but Code Not Generated | 🟠 Medium | `ipcHandlers.ts`, `IntelligenceEngine.ts` | | ||
| | [Bug 003](./bug-003-skill-injection-failure.md) | Skill Injection Ignored in V3 Engine | 🟠 Medium | `WhatToAnswerLLM.ts` | | ||
|
|
||
| --- | ||
|
|
||
| ## Blast Radius Summary (jcodemunch analysis) | ||
|
|
||
| All three bugs touch the core answering loop. Risk is **concentrated** in a small set of files | ||
| but the flag `documentGroundedCustomModeActive` fans out to 26 dependent files. | ||
|
|
||
| ``` | ||
| documentGroundedCustomModeActive (AnswerPlanner.ts) | ||
| ├── WhatToAnswerLLM.ts [6 refs — also affected by Bug 003] | ||
| ├── contextRoute.ts [5 refs — secondary fix site for Bug 001] | ||
| ├── ModesManager.ts [8 refs] | ||
| ├── conversationHistoryPolicy.ts [1 ref] | ||
| └── modeProfiles.ts [1 ref] | ||
|
|
||
| buildCustomModeExecutionContract | ||
| ├── IntelligenceEngine.ts [5 refs — fix site for Bug 002] | ||
| └── ipcHandlers.ts [6 refs — fix site for Bug 002] | ||
|
|
||
| WhatToAnswerLLM (class) | ||
| └── llm/index.ts [2 refs] | ||
| ``` | ||
|
|
||
| > **⚠️ All confirmed dependents have `has_test_reach: false`** — meaning none of the above files | ||
| > are currently covered by a test that verifies end-to-end answering behaviour. | ||
| > Each bug doc includes a checklist of tests to add or update. | ||
|
|
||
| --- | ||
|
|
||
| ## Key Additional Finding (Bug 001) | ||
|
|
||
| `documentGroundedFromContract()` in [`modeSourceContract.ts`](../../electron/services/modeSourceContract.ts) | ||
| **already implements the correct guard** (`if (!hasReferenceFiles) return false`). | ||
| It has **zero call-sites** in the entire codebase. | ||
|
|
||
| The recommended fix for Bug 001 is to wire this function in rather than adding a new inline check. | ||
|
|
||
| --- | ||
|
|
||
| ## How to Review | ||
|
|
||
| 1. Read each bug file in [`docs/bugs/`](./) for full root-cause analysis, diffs, and safe-change checklists. | ||
| 2. Fixes are **surgical** — each change is a single guard condition or a 3-line append. | ||
| 3. Run `ModePolicyShadowDivergence2026_07_26.test.mjs` and `Issue303SkillInvocation.test.mjs` | ||
| after implementing fixes to catch regressions early. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,142 @@ | ||
| # Bug 001 — Document-Grounded Refusal on General / YouTube Questions | ||
|
|
||
| ## User Reports (Telegram) | ||
|
|
||
| > *"I just asked a random question from a YouTube, and it answers: 'This is not directly uploaded document.'"* | ||
| > *"I noticed that instead of generating an answer based on the conversation context, it keeps saying 'no document has been uploaded.'"* | ||
|
|
||
| --- | ||
|
|
||
| ## Root Cause | ||
|
|
||
| ### 1. `documentGroundedCustomModeActive` ignores `hasReferenceFiles` | ||
|
|
||
| **File:** [`electron/llm/AnswerPlanner.ts`](../../electron/llm/AnswerPlanner.ts) — line 2335 | ||
|
|
||
| ```typescript | ||
| // CURRENT (broken) | ||
| const documentGroundedCustomModeActive = | ||
| input.activeMode?.documentGroundedCustomModeActive === true; | ||
| ``` | ||
|
|
||
| This reads the pre-computed flag directly from the mode object. The flag is set by `ModesManager` for any mode whose `sourceAuthority` is `reference_files_primary` (or similar) — **even when no files have been uploaded**. | ||
|
|
||
| ### 2. The safe guard function exists but is never called | ||
|
|
||
| **File:** [`electron/services/modeSourceContract.ts`](../../electron/services/modeSourceContract.ts) — line 730 | ||
|
|
||
| ```typescript | ||
| function documentGroundedFromContract( | ||
| contract: ModeSourceContract, | ||
| hasReferenceFiles: boolean, // ← correctly short-circuits | ||
| ): boolean { | ||
| if (!hasReferenceFiles) return false; // ← guard we need | ||
| return contract.sourceAuthority === 'reference_files_only' | ||
| || contract.sourceAuthority === 'reference_files_primary' | ||
| || contract.sourceAuthority === 'reference_files_plus_transcript'; | ||
| } | ||
| ``` | ||
|
|
||
| > **This function has 0 call-sites in the entire codebase** (confirmed via `find_references`). | ||
| > It was built to solve exactly this problem but was never wired in. | ||
|
|
||
| ### 3. Forced document-grounded routing with empty context | ||
|
|
||
| **File:** [`electron/llm/AnswerPlanner.ts`](../../electron/llm/AnswerPlanner.ts) — line 2822 | ||
|
|
||
| ```typescript | ||
| if (documentGroundedCustomModeActive && !explicitDocumentModeCodingAsk && !explicitDocumentModeProfileAsk) { | ||
| const docShape = classifyDocumentQuestionShape(question, ...); | ||
| answerType = docShape === 'broad_overview' ? 'lecture_answer' : docShape; | ||
| } | ||
| ``` | ||
|
|
||
| Because the flag is `true` (incorrectly), the turn is classified as `lecture_answer` / `definitional_answer` / etc. | ||
|
|
||
| ### 4. Validation fails against empty `docContextBlock` | ||
|
|
||
| **Files:** [`electron/ipcHandlers.ts`](../../electron/ipcHandlers.ts) (line 4029) and [`electron/IntelligenceEngine.ts`](../../electron/IntelligenceEngine.ts) (line 3156) | ||
|
|
||
| The WTA gate checks that a non-empty `docContextBlock` exists for document-grounded answer types. Since no files were uploaded, the block is empty → validation fails → refusal message returned. | ||
|
|
||
| --- | ||
|
|
||
| ## Also Affected: `contextRoute.ts` | ||
|
|
||
| **File:** [`electron/llm/contextRoute.ts`](../../electron/llm/contextRoute.ts) — line 76 | ||
|
|
||
| ```typescript | ||
| // Same pattern, same bug: | ||
| const documentGroundedCustomModeActive = plan.documentGroundedCustomModeActive === true; | ||
| ``` | ||
|
|
||
| This copy of the flag is used for conversation-history routing and WhatToAnswerLLM prompt shaping — it inherits the same incorrect `true` value. | ||
|
|
||
| --- | ||
|
|
||
| ## Proposed Fix | ||
|
|
||
| ### Fix A — `AnswerPlanner.ts` (primary routing gate) | ||
|
|
||
| ```diff | ||
| - const documentGroundedCustomModeActive = | ||
| - input.activeMode?.documentGroundedCustomModeActive === true; | ||
| + const hasReferenceFiles = input.activeMode?.hasReferenceFiles === true; | ||
| + const documentGroundedCustomModeActive = | ||
| + input.activeMode?.documentGroundedCustomModeActive === true && hasReferenceFiles; | ||
| ``` | ||
|
|
||
| **Alternative (preferred):** wire in the already-existing safe utility: | ||
|
|
||
| ```diff | ||
| + import { documentGroundedFromContract } from '../services/modeSourceContract'; | ||
| // … | ||
| const documentGroundedCustomModeActive = | ||
| - input.activeMode?.documentGroundedCustomModeActive === true; | ||
| + documentGroundedFromContract( | ||
| + input.activeMode?.sourceContract, | ||
| + input.activeMode?.hasReferenceFiles === true, | ||
| + ); | ||
| ``` | ||
|
|
||
| ### Fix B — `contextRoute.ts` (conversation-history / WTA routing) | ||
|
|
||
| Apply the same guard so that the `plan` object propagated to WhatToAnswerLLM is consistent: | ||
|
|
||
| ```diff | ||
| - const documentGroundedCustomModeActive = plan.documentGroundedCustomModeActive === true; | ||
| + const documentGroundedCustomModeActive = | ||
| + plan.documentGroundedCustomModeActive === true && (plan.hasReferenceFiles === true); | ||
| ``` | ||
|
|
||
| --- | ||
|
|
||
| ## Blast Radius | ||
|
|
||
| | Symbol | Confirmed Dependents | Potential (namespace import) | | ||
| |---|---|---| | ||
| | `documentGroundedCustomModeActive` (AnswerPlanner) | 5 files | 21 files | | ||
| | `classifyDocumentQuestionShape` | 3 files | 6 files | | ||
| | `buildCustomModeExecutionContract` | 2 files | 4 files | | ||
|
|
||
| **High-blast files that must be regression-tested after this fix:** | ||
| - `electron/llm/WhatToAnswerLLM.ts` (6 references to the flag) | ||
| - `electron/llm/contextRoute.ts` (5 references) | ||
| - `electron/services/ModesManager.ts` (8 references) | ||
| - `electron/llm/conversationHistoryPolicy.ts` | ||
| - `electron/llm/modeProfiles.ts` | ||
|
|
||
| **Files with no test coverage flagged by jcodemunch:** | ||
| - All confirmed dependents listed above have `has_test_reach: false` | ||
| - Recommendation: add a dedicated Jest test for the `documentGroundedFromContract` path with `hasReferenceFiles: false` | ||
|
|
||
| --- | ||
|
|
||
| ## Safe-Change Checklist | ||
|
|
||
| - [ ] Fix A applied in `AnswerPlanner.ts` | ||
| - [ ] Fix B applied in `contextRoute.ts` | ||
| - [ ] `documentGroundedFromContract` verified to return `false` when `hasReferenceFiles` is falsy | ||
| - [ ] Manual smoke-test: open "General" mode, ask a question, verify no refusal | ||
| - [ ] Existing doc-grounded tests still pass when files are uploaded | ||
| - [ ] No regression in `ModePolicyShadowDivergence2026_07_26.test.mjs` (tests the `documentGroundedCustomModeActive` flag parity) | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,122 @@ | ||
| # Bug 002 — Screenshot Attached but Code Not Generated | ||
|
|
||
| ## User Report (Telegram) | ||
|
|
||
| > *"when i take a screenshot and chat shows screenshot attached, and i ask to give me the code, it doesn't give me the code"* | ||
|
|
||
| --- | ||
|
|
||
| ## Root Cause Chain | ||
|
|
||
| ### 1. `hasScreenContext` omitted in manual-chat IPC handler | ||
|
|
||
| **File:** [`electron/ipcHandlers.ts`](../../electron/ipcHandlers.ts) — inside `_geminiChatStreamHandler` (~line 1163) | ||
|
|
||
| When `buildV3Prompt` is called for the `gemini-chat-stream` IPC path, `hasScreenContext` is **completely absent** from the argument object: | ||
|
|
||
| ```typescript | ||
| const composed = await buildV3Prompt({ | ||
| surface: 'manual-chat', | ||
| pathTag: 'ipc', | ||
| question: String(message || ''), | ||
| modeTemplateType: rawMode, | ||
| modeUniqueId: modeInfo?.id ?? null, | ||
| attachedSourceCount: files.length, | ||
| // … other fields … | ||
| // hasScreenContext: ← MISSING | ||
| }); | ||
| ``` | ||
|
|
||
| The field defaults to `undefined` / `false` inside `buildV3Prompt`. | ||
|
|
||
| ### 2. `IntelligenceEngine` only checks `options.screenContext`, not `imagePaths` | ||
|
|
||
| **File:** [`electron/IntelligenceEngine.ts`](../../electron/IntelligenceEngine.ts) — inside `runWhatShouldISay` (~line 2494) | ||
|
|
||
| ```typescript | ||
| // CURRENT (broken): | ||
| hasScreenContext: Boolean(options?.screenContext), | ||
| ``` | ||
|
|
||
| `options.screenContext` is the **OCR object** built from the periodic screen-capture service — it is not set for manually-attached screenshots sent through `imagePaths`. When a user attaches a screenshot through the chat UI, `imagePaths` is populated but `screenContext` is `null`, so `hasScreenContext` is `false`. | ||
|
|
||
| ### 3. Turn classifier misses `SCREEN_SPECIFIC` / `SCREEN_FACT` | ||
|
|
||
| **File:** [`electron/context-intelligence/question/turn-classifier.ts`](../../electron/context-intelligence/question/turn-classifier.ts) | ||
|
|
||
| The classifier uses `hasScreenContext` to mark turns as screen-anchored. With `hasScreenContext: false`, a query like *"give me the code"* receives no screen-related label. | ||
|
|
||
| ### 4. Coding persona never activated | ||
|
|
||
| **File:** [`electron/llm/AnswerPlanner.ts`](../../electron/llm/AnswerPlanner.ts) | ||
|
|
||
| Without a `SCREEN_SPECIFIC` or `CODING_TASK_RE` match, the orchestrator falls through to a generic `general_meeting_answer` type and never activates the coding persona or code-extraction prompt. | ||
|
|
||
| > **Note:** the phrase *"give me the code"* does not match `CODING_TASK_RE` (which requires verbs like write / implement / solve). The primary fix must therefore be at the `hasScreenContext` propagation level, not keyword-tuning. | ||
|
|
||
| --- | ||
|
|
||
| ## Proposed Fixes | ||
|
|
||
| ### Fix A — `electron/ipcHandlers.ts` | ||
|
|
||
| Inside `_geminiChatStreamHandler`, pass `hasScreenContext` from `imagePaths`: | ||
|
|
||
| ```diff | ||
| const composed = await buildV3Prompt({ | ||
| surface: 'manual-chat', | ||
| pathTag: 'ipc', | ||
| question: String(message || ''), | ||
| modeTemplateType: rawMode, | ||
| modeUniqueId: modeInfo?.id ?? null, | ||
| attachedSourceCount: files.length, | ||
| attachedFileNames: (files as Array<{ fileName?: string }>) | ||
| .map((f) => f.fileName ?? '').filter(Boolean), | ||
| profileSourceCount: v3ProfileCounts.profileResume + ..., | ||
| resolvedProfileSources: v3ProfileResolved, | ||
| extraAllowedSourceTypes: extraSourceTypes, | ||
| debugSources: v3DebugSources as never, | ||
| deferDebugCompletion: true, | ||
| requestId: `v3-${myStreamId}`, | ||
| requestSequence: myStreamId, | ||
| + hasScreenContext: Boolean(imagePaths && imagePaths.length > 0), | ||
| }); | ||
| ``` | ||
|
|
||
| ### Fix B — `electron/IntelligenceEngine.ts` | ||
|
|
||
| Expand the `hasScreenContext` guard to cover the `imagePaths` channel: | ||
|
|
||
| ```diff | ||
| - hasScreenContext: Boolean(options?.screenContext), | ||
| + hasScreenContext: Boolean( | ||
| + options?.screenContext || (imagePaths && imagePaths.length > 0) | ||
| + ), | ||
| ``` | ||
|
|
||
| --- | ||
|
|
||
| ## Blast Radius | ||
|
|
||
| | Symbol / File | Confirmed Dependents | Risk | | ||
| |---|---|---| | ||
| | `ScreenContextService` | `IntelligenceEngine`, `IntelligenceManager`, `WhatToAnswerLLM` | Low — existing callers only read `captureScreen()`, not `imagePaths` | | ||
| | `buildCustomModeExecutionContract` | `IntelligenceEngine` (×5), `ipcHandlers` (×6) | Medium — touches both primary entry points | | ||
| | `ipcHandlers.ts` change | Only `gemini-chat-stream` handler | Low — scoped to manual-chat surface | | ||
| | `IntelligenceEngine.ts` change | All `runWhatShouldISay` callers | Medium — also affects overlay auto-answer flow | | ||
|
|
||
| **Regression risk for overlay auto-answer:** | ||
| The overlay path sends both `options.screenContext` (OCR object) and `imagePaths`. After Fix B, both still resolve to `true` — no behaviour change for the existing overlay path. | ||
|
|
||
| **Test file to update:** | ||
| - [`electron/services/__tests__/IntelligenceEngineScreenContext.test.mjs`](../../electron/services/__tests__/IntelligenceEngineScreenContext.test.mjs) — add a case where `screenContext` is `null` but `imagePaths` is non-empty, assert `hasScreenContext === true`. | ||
|
|
||
| --- | ||
|
|
||
| ## Safe-Change Checklist | ||
|
|
||
| - [ ] Fix A applied in `ipcHandlers.ts` | ||
| - [ ] Fix B applied in `IntelligenceEngine.ts` | ||
| - [ ] Manual smoke-test: attach a screenshot in manual chat, type "give me the code", verify coding response | ||
| - [ ] Overlay auto-answer with screen capture still works (no regression) | ||
| - [ ] `IntelligenceEngineScreenContext.test.mjs` updated with new `imagePaths`-only case |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.