Fix blocked webpage fetches with a requests fallback

[LIMINGNING]

Problem

fetch_page_content_async can fail on pages that remain reachable through the synchronous requests extractor. In the reproduced Wikipedia case, the aiohttp path timed out or received a blocked response while requests returned HTTP 200. When WebParserClient_url was not configured, the async path had no second retrieval channel and downstream code received only a search-result snippet. The short/empty-response branch also referenced an undefined exception variable.

Changes

• Select request headers per URL and send a descriptive user agent to Wikimedia domains.
• Retry HTTP errors, short blocked responses, and async exceptions through the existing synchronous extractor.
• Run the synchronous fallback with asyncio.to_thread so it does not block the event loop.
• Replace the undefined exception reference with a deterministic empty/blocked-response message.
• Add focused tests for Wikimedia headers, ordinary-domain headers, off-event-loop execution, and async failure recovery.

The fallback is generic and contains no task-, domain-content-, or answer-specific logic.

Verification

• python -m unittest discover -s scripts/tests -p "test_*.py": 4 tests passed.
• Live smoke test against https://en.wikipedia.org/wiki/Mercedes_Sosa: retrieved 20,000 characters and found the expected Studio albums section and its 2000-2009 entries.
• The branch is based directly on the current upstream main revision (db387eb).

Scope

This PR changes only scripts/search/bing_search.py and adds scripts/tests/test_web_fetch_fallback.py. It does not modify model prompts, search-query generation, API configuration, or answer logic.

[Copilot]

Pull request overview

This PR improves resilience of fetch_page_content_async by adding a non-blocking (threaded) synchronous requests fallback when the aiohttp path fails (HTTP errors, blocked/short responses, or exceptions), and by selecting request headers per-domain (notably a Wikimedia-specific user agent).

Changes:

• Added request_headers_for_url() and a Wikimedia-specific User-Agent.
• Implemented fetch_with_requests_fallback() using asyncio.to_thread() and wired it into extract_text_from_url_async() error paths.
• Added unit tests covering header selection and fallback behavior.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File	Description
scripts/search/bing_search.py	Adds per-URL header selection, and introduces a threaded synchronous fallback for async fetch failures/blocked responses.
scripts/tests/test_web_fetch_fallback.py	Adds tests for Wikimedia vs non-Wikimedia headers and for async-to-sync fallback execution.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.