A benchmark-first way to choose DeerFlow's web search provider

[auxiliar-ag]

DeerFlow's setup wizard can configure an optional web search provider, and its tool layer is designed to be extended or replaced. That flexibility raises a practical question: which search provider should a deployment start with?

I would not turn the first row of a public leaderboard into a production default. A ranking compresses several measurements into one number, while a DeerFlow workload still has its own priorities. An interactive research session may care about response time. A long-running research task may favor coverage. A scheduled batch may accept another second of latency when useful results cost less.

The better use of a public benchmark is to narrow the field before testing a smaller sample of your own queries.

Read the raw columns before the composite

I started with a public benchmark of 11 web search providers. The providers ran the same dated corpus, and the table publishes composite score, recall@10, median latency, cost per useful result, and the run date.

The top rows show why the raw measurements matter:

Provider	Composite	Recall@10	Cost per useful result	Median latency
Jina	7.96	0.62	$0.00032	1.0 s
Serper	7.79	0.62	$0.00048	816 ms
You.com	7.60	0.61	$0.0082	566 ms
SerpApi	7.42	0.59	$0.015	181 ms

Those measurements do not identify one universal winner. Jina and Serper had the same recall in this run, with different cost and latency. You.com returned faster results with similar recall at a higher measured cost. SerpApi was much faster, but recall and cost moved in the other direction.

The composite helps with scanning. It does not remove the tradeoff.

Start with the DeerFlow task

Before choosing a provider, write down what would make the search step fail for the task you expect DeerFlow to perform.

For an interactive agent, a slow response may consume most of the turn budget. For a monitoring or research job, the same latency may be acceptable while missing a relevant source is not. For a high-volume workflow, cost per useful result can matter more than price per call because failed or irrelevant calls still consume money and time.

This produces three simple shortlist patterns:

1. A latency-sensitive workflow starts with the faster candidates that still meet its minimum result quality.
2. A coverage-sensitive workflow begins with the strongest recall candidates, then checks whether their result mix matches the workload.
3. A cost-sensitive batch compares cost per useful result rather than nominal request price.

These are shortlist rules, not routing rules.

Build a small local corpus

You do not need to reproduce a large public benchmark. Start with 20 to 50 queries sampled from the work your DeerFlow deployment will perform.

Include cases that are likely to break:

• recent events where freshness matters;
• niche technical queries with few authoritative sources;
• ambiguous terms that need context;
• queries with domain or language constraints;
• questions where missing one source changes the answer.

Define the expected result before running the comparison. For search, that can be a set of relevant URLs, domains, or documents. Record latency and cost alongside recall. Keep errors in the denominator so a cheap failed call does not look efficient.

Run the same queries against two or three providers from the public shortlist. This local sample answers the question the public table cannot: which provider fits your traffic and tasks?

Add fallback only when failures differ

A second provider is useful when it covers a failure mode the first one does not. It is not useful merely because both services expose a search endpoint.

Compare misses at the query level. If both providers fail on the same queries, fallback adds cost without adding coverage. If their misses differ, the second provider may recover useful results.

Keep the routing condition explicit. Retry on transport failure, rate limiting, or a response that violates the task contract. Do not send every query to every provider unless the workload needs that level of coverage.

An attempt receipt can record the provider, latency, validation result, and fallback reason. Avoid putting private query content into high-cardinality analytics fields.

Recheck instead of freezing the winner

Provider behavior changes. Pricing changes, indexes evolve, and a corpus becomes stale. Record the benchmark date next to the decision and rerun the local sample on a schedule that matches the importance of the workflow.

A public leaderboard can tell you where to start. The queries your DeerFlow deployment actually runs should decide what ships.

This is not an official DeerFlow integration or a benchmark of DeerFlow itself. It is a provider-selection process for deployments that enable web search.

Disclosure: I am affiliated with NativePort, which produced the benchmark linked above. The measurements and run date are public. This post was prepared with AI assistance and reviewed before publication.