Studio: Stop Deep Research deleting links from code in its report - #10814
Studio: Stop Deep Research deleting links from code in its report#10814NilayYadav wants to merge 6 commits into
Conversation
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 338407365f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if fence is None or (fence.group(1)[0] == "`" and "`" in content[fence.end() :]): | ||
| pieces.append(_INLINE_CODE.sub(lambda match: mask(match.group(0)), line)) |
There was a problem hiding this comment.
When the model emits a valid four-space- or tab-indented code block instead of a fence, this branch treats every line as prose. For example, an indented curl https://private.example/api becomes just curl, while print(x[1]) can have [1] rewritten as a citation, so the copied code remains corrupted despite this fix. Detect and mask indented code blocks as well.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 5c7506e: indented and list code blocks are masked from markdown-it block maps. Covered by test_report_preserves_indented_and_list_code.
| for line in text.splitlines(keepends = True): | ||
| content = line.rstrip("\r\n") | ||
| fence = _MARKDOWN_FENCE.match(content) | ||
| if opening is None: | ||
| if fence is None or (fence.group(1)[0] == "`" and "`" in content[fence.end() :]): | ||
| pieces.append(_INLINE_CODE.sub(lambda match: mask(match.group(0)), line)) |
There was a problem hiding this comment.
Markdown code spans may contain line endings, but applying _INLINE_CODE independently to each split line means neither delimiter is matched. With a span such as Use \curl\nhttps://private.example/api` now`, the URL is still removed from the stored report. Scan inline spans across the complete non-fenced text rather than one line at a time.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 5c7506e: code spans are parsed over the whole inline block, so multiline spans are masked as one unit. Covered by test_multiline_inline_code_keeps_urls_and_brackets.
| if fence is None or (fence.group(1)[0] == "`" and "`" in content[fence.end() :]): | ||
| pieces.append(_INLINE_CODE.sub(lambda match: mask(match.group(0)), line)) |
There was a problem hiding this comment.
When prose uses escaped backticks to display literal backticks, such as \https://nope.example\``, _INLINE_CODE still recognizes the pair because it ignores backslash escaping. _mask_code consequently restores the unsupported URL verbatim instead of letting _RAW_URL remove it, allowing ordinary rendered prose to bypass gathered-source validation.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 5c7506e: spans come from the CommonMark backtick rule, which honours backslash escapes. Covered by test_literal_escaped_backticks_do_not_hide_prose_urls.
| if opening is not None: | ||
| pieces.append(mask(text[opening:])) |
There was a problem hiding this comment.
When an unclosed fenced block is nested in a block quote or list, Markdown ends that block when the containing quote/list ends, but opening remains set here until a matching fence or EOF. Consequently, ordinary following prose such as Outside https://nope.example and [1] is masked and restored unchanged, allowing unsupported URLs and citations to bypass validation. Track the opening container and stop masking when it ends.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 5c7506e: a fence now ends where markdown-it ends its container. Covered by test_unclosed_quote_fence_stops_at_the_quote_boundary.
| source_urls = list(source_by_url) | ||
| placeholders: dict[str, str] = {} | ||
|
|
||
| report = _mask_code(report, placeholders) |
There was a problem hiding this comment.
Masking code only inside _validate_report_sources does not preserve it in the delivered report because research_runs.py immediately passes the restored result to _validate_report_document_sources. That validator still removes document-shaped text inside recognized code; for example, pattern = "[Document: generated]" in a fenced Python block becomes pattern = "". Keep code masked through both validators or make the document validator code-aware.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 5c7506e: _validate_report_document_sources masks code too. Covered by test_delivered_report_keeps_document_literals_in_code.
| source_urls = list(source_by_url) | ||
| placeholders: dict[str, str] = {} | ||
|
|
||
| report = _mask_code(report, placeholders) |
There was a problem hiding this comment.
The production path immediately passes this result to _validate_report_document_sources, after these placeholders have already been restored. Therefore even a correctly recognized fenced block is still changed when it contains code such as pattern = "[Document: generated]": the second validator removes that substring and stores pattern = "". Keep code masked through both validators or make document code in the document-citation validator too.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Same as the item above; fixed in 5c7506e.
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5ddaf72bef
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if token.type in {"fence", "code_block"}: | ||
| # Keep the following heading on its own line while the block is masked. | ||
| end = start + len(text[start:end].rstrip("\r\n")) | ||
| spans.append((start, end)) | ||
| elif token.type == "inline": |
There was a problem hiding this comment.
When a report uses valid HTML code markup rather than Markdown fences/backticks, these branches ignore the resulting html_block/html_inline content. For example, <pre><code>curl https://private.example/api</code></pre> is still rewritten to <pre><code>curl </code></pre>, and bracketed expressions inside <code> can still become citations. Mask <pre>/<code> contents as code too so saved and copied examples are not corrupted.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Not changing this here. The synthesis prompt asks for Markdown, and HTML
/ content is rewritten the same way on main, so this PR does not introduce it. Masking raw HTML elements would also let arbitrary HTML skip link validation, so it needs its own change with bypass tests.
|
@codex review |
1 similar comment
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5ddaf72bef
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| # Tokenize valid citations first so a "]" inside a filename ("budget [final].pdf") does not | ||
| # truncate them, then strip the invalid ones and restore the valid. | ||
| placeholders: dict[str, str] = {} | ||
| report = _mask_code(report, placeholders) |
There was a problem hiding this comment.
When _validate_report_sources removes an unsupported URL that was the only content on a line, reparsing here can change the Markdown structure before document validation. For example, Context\nhttps://nope.example\n [Document: hallucinated] initially contains one paragraph, but the production sequence in core/research_runs.py turns it into Context\n\n [Document: hallucinated]; this call then classifies the indented citation as a code block and masks it, allowing the unsupported document citation to survive. Preserve the code spans identified from the original draft across both validators rather than deriving new spans after URL rewriting.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 13ac9d4: the draft is masked once and both validators run on that masked text (_validate_report), so removing a URL-only line can no longer turn an indented citation into code. Covered by test_removed_url_line_does_not_turn_a_document_citation_into_code.
Review, GitHub Actions and Studio UI evidencePinned head: GitHub Actions A/B Both runs use the same workflow, probe and
Studio UI, before and after The run goes through the real Highlighted lines differ between the two installs: red on
Both sides rendered 3 code blocks, and neither Studio log contains a traceback. Review fixes pushed
Tests
CI
|
|
@codex review |
|
Codex Review: Didn't find any major issues. 🚀 Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
|
Confirmed this hits studio/backend/core/research/citations.py, where the link and citation pass still runs over fenced and inline code, so saved commands lose their URLs. Will get this reviewed. |
|
Deep Research citation validation now keeps command URLs, Python indexes, and citation-like literals in code. On All 442 affected tests passed, with parser checks on markdown-it-py 3.0 and 4.2. Both Windows installations completed successfully, and the upgrade and rollback preserved the saved chat. Five optional CI jobs remain red: the sidebar assertion also fails on the base, the DeepSeek timing failure did not reproduce locally, and package scans flag scikit-learn and unsloth-zoo. No required checks are reported. The image uses a deterministic report fixture without live-provider inference. Native browser coverage is Edge; non-Windows compatibility is reasoned. Before:
|
3 participants
Deep Research removed web addresses from the code and commands in its reports. A git clone command lost its GitHub link, pip install lost its --index-url address, and base_url="http://localhost:8888/v1" became base_url=". Code like x.shape[1] was also turned into a link to a source. The broken text was saved as the report and shown in the chat, so copied commands did not work.
This happened because the step that removes unverified links and numbered citations ran over the whole report, including code. Now code blocks and inline code are set aside during that step and put back exactly as written. A Sources heading inside a code block also no longer cuts off the rest of the report.
Links and citations in normal text work as before. Tested with a full Deep Research run. Before, the saved report had the broken commands above. After, every command and code line was saved unchanged.