feat(web): add Serply as an optional backend for web and Scholar search - #322
Open
googio wants to merge 1 commit into
Open
feat(web): add Serply as an optional backend for web and Scholar search#322googio wants to merge 1 commit into
googio wants to merge 1 commit into
Conversation
Serply (https://serply.io) returns Google web and Google Scholar results over a keyed REST API. When SERPLY_API_KEY (or web_search.serply_api_key) is set: - WebSearchClient tries Serply after Tavily and before the DuckDuckGo HTML scrape. - GoogleScholarClient queries Serply's Scholar endpoint first and only falls back to scholarly scraping if that call fails. This also lets the Scholar client work without the optional scholarly package. Uses urllib only, no new dependency. Without a Serply key nothing changes. Config gains serply_api_key / serply_api_key_env, which the literature stage passes through to WebSearchAgent.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds Serply as an optional keyed backend in the web layer, following the pattern Tavily already uses (key from config or env, nothing changes without it):
WebSearchClienttries Serply's/v1/search/(Google web results) after Tavily and before the DuckDuckGo HTML fallback.GoogleScholarClientqueries Serply's/v1/scholar/first when a key is set and falls back toscholarlyif that call fails. The client can now be constructed withoutscholarlyinstalled as long as a Serply key is present.WebSearchConfiggainsserply_api_key/serply_api_key_env(defaultSERPLY_API_KEY);_literature.pypasses the resolved key toWebSearchAgentthe same way it does for Tavily.Plain
urllibonly, no new dependency.Why
Refs #321. There is currently no keyed option for Google web or Google Scholar results, and Scholar results require the
scholarlyscraper, which Google blocks on unattended runs. Serply gives Stage 4's web augmentation a stable source for both, and lets Scholar work without the optionalwebextra.Notes
SERPLY_API_KEYis unset.User-Agent; Serply's edge rejects requests without one.descriptionline ("A Author, B Author - Venue, 2021"); a trailing, YYYYis stripped so venues such as2022 IEEE/CVF Conference ... (CVPR), 2022keep their name.tests/test_web_search.py(TestSerplyBackend),tests/test_web_scholar.py(TestSerplyScholarBackend) andtests/test_rc_config.py;tests/test_rc_executor.pyextended for the new kwarg. Network is mocked.web_searchexample gains theserply_api_key_envline. Translated READMEs underdocs/were left alone.Testing
Live check with a real
SERPLY_API_KEYandscholarlynot installed:WebSearchClient(serply_api_key=...).search("knowledge distillation survey 2024")returned 5 results (sourceserply, 1.30s);GoogleScholarClient(serply_api_key=...).search(..., limit=5)returned 5 papers with authors, year, venue and citation counts (first:gou2021knowledge, OpenAlex idW3034368386);WebSearchAgent(serply_api_key=...)produced 3 web and 3 scholar results. With no keys set, the client goes to the DuckDuckGo path exactly as before.Disclosure: I work with Serply. Happy to adjust scope or naming, or drop this
entirely if it isn't a direction you want for the project.