Skip to content

feat(web): add Serply as an optional backend for web and Scholar search - #322

Open
googio wants to merge 1 commit into
aiming-lab:mainfrom
googio:feat/serply-search
Open

feat(web): add Serply as an optional backend for web and Scholar search#322
googio wants to merge 1 commit into
aiming-lab:mainfrom
googio:feat/serply-search

Conversation

@googio

@googio googio commented Aug 25, 2026

Copy link
Copy Markdown

What

Adds Serply as an optional keyed backend in the web layer, following the pattern Tavily already uses (key from config or env, nothing changes without it):

  • WebSearchClient tries Serply's /v1/search/ (Google web results) after Tavily and before the DuckDuckGo HTML fallback.
  • GoogleScholarClient queries Serply's /v1/scholar/ first when a key is set and falls back to scholarly if that call fails. The client can now be constructed without scholarly installed as long as a Serply key is present.
  • WebSearchConfig gains serply_api_key / serply_api_key_env (default SERPLY_API_KEY); _literature.py passes the resolved key to WebSearchAgent the same way it does for Tavily.

Plain urllib only, no new dependency.

Why

Refs #321. There is currently no keyed option for Google web or Google Scholar results, and Scholar results require the scholarly scraper, which Google blocks on unattended runs. Serply gives Stage 4's web augmentation a stable source for both, and lets Scholar work without the optional web extra.

Notes

  • Serply support is optional, and behavior is unchanged when SERPLY_API_KEY is unset.
  • Requests carry an explicit User-Agent; Serply's edge rejects requests without one.
  • Venue and year for Scholar results are parsed from Serply's description line ("A Author, B Author - Venue, 2021"); a trailing , YYYY is stripped so venues such as 2022 IEEE/CVF Conference ... (CVPR), 2022 keep their name.
  • Tests added in tests/test_web_search.py (TestSerplyBackend), tests/test_web_scholar.py (TestSerplyScholarBackend) and tests/test_rc_config.py; tests/test_rc_executor.py extended for the new kwarg. Network is mocked.
  • README web_search example gains the serply_api_key_env line. Translated READMEs under docs/ were left alone.

Testing

$ env -u SERPLY_API_KEY -u TAVILY_API_KEY .venv/bin/python -m pytest tests/test_web_search.py tests/test_web_scholar.py tests/test_rc_config.py -q
................................................................         [100%]
64 passed in 0.27s

$ env -u SERPLY_API_KEY -u TAVILY_API_KEY .venv/bin/python -m pytest tests/ -q --ignore=tests/e2e_docker_sandbox.py --ignore=tests/e2e_real_llm.py
2965 passed, 56 skipped, 1 warning in 82.23s (0:01:22)

Live check with a real SERPLY_API_KEY and scholarly not installed: WebSearchClient(serply_api_key=...).search("knowledge distillation survey 2024") returned 5 results (source serply, 1.30s); GoogleScholarClient(serply_api_key=...).search(..., limit=5) returned 5 papers with authors, year, venue and citation counts (first: gou2021knowledge, OpenAlex id W3034368386); WebSearchAgent(serply_api_key=...) produced 3 web and 3 scholar results. With no keys set, the client goes to the DuckDuckGo path exactly as before.

Disclosure: I work with Serply. Happy to adjust scope or naming, or drop this
entirely if it isn't a direction you want for the project.

Serply (https://serply.io) returns Google web and Google Scholar results
over a keyed REST API. When SERPLY_API_KEY (or web_search.serply_api_key)
is set:

- WebSearchClient tries Serply after Tavily and before the DuckDuckGo
  HTML scrape.
- GoogleScholarClient queries Serply's Scholar endpoint first and only
  falls back to scholarly scraping if that call fails. This also lets
  the Scholar client work without the optional scholarly package.

Uses urllib only, no new dependency. Without a Serply key nothing
changes. Config gains serply_api_key / serply_api_key_env, which the
literature stage passes through to WebSearchAgent.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant