Skip to content

feat: Add harness evaluator for retrieval pipeline - #3947

Merged
sjrl merged 14 commits into
mainfrom
agent-pack/retrieval-harness-evaluator
Sep 25, 2026
Merged

sjrl merged 14 commits into
mainfrom
agent-pack/retrieval-harness-evaluator

Conversation

@sjrl

@sjrl sjrl commented Sep 11, 2026 •

Copy link
Copy Markdown
Contributor

Related Issues

Proposed Changes:

This PR is another reviewable slice of the larger harness evaluator/optimization work in #3932.

It introduces HarnessEvaluator, which evaluates an Agent or Pipeline against a set of eval cases and returns normalized EvaluationMetrics, including quality, latency, per-model token usage, and evaluator-specific details.

The first implementation, RetrievalHarnessEvaluator, evaluates retrieval pipelines by discovering query inputs and a single documents output from socket names rather than component names. This lets the same evaluator work with anything from a simple BM25 retriever to a multi-stage retrieval/reranking pipeline. validate() rejects unsupported pipelines before evaluation starts.

Each RetrievalEvalCase contains a question and the expected evidence by document ID. Cases are scored with recall@k, precision@k, and found_at. Overall quality is based on recall@k, so partial retrieval improvements are reflected in the score.

Supporting pieces included in this PR:

  • RetrievalEvalCase and retrieval scoring utilities.
  • haystack_integrations.tracing.agent_pack, a lightweight tracer used to collect per-model token usage, per-component output sizes, and small socket samples.
  • ComponentLogCollector for capturing warnings emitted by components.
  • examples/retrieval_eval.py, which evaluates a BM25 pipeline over the MultiHopRAG corpus and reports aggregate and per evaluation case results.

Here is what running the new retrieval_eval.py example file produces

=== 1. set up corpus and evaluation set ===
Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
  multihop-rag in memory: 4096 chunks from 609 articles
  eval cases: 20 labelled from evidence, scored at recall@5

=== 2. evaluate ===
  pipeline: expander -> retriever
  gpt-5.4 expanding into 3 extra queries, 10 documents retrieved per query

=== 3. outcome ===
  quality=0.35 (mean recall@5)  recall@k=0.35  precision@k=0.19  latency=2698ms
    chat_generator.replies: 1 typical (1-1)
    expander.queries: 4 typical (4-4)
    retriever.documents: 17 typical (13-30)
    gpt-5.4-2026-03-05: 7279 in + 2584 out (493 per eval case)

=== eval case 1/20: FAIL ===
  question: Did "The Independent - Life and Style" article on "Jada Pinkett Smith and Will Smith's marriage" disagree with the same publication's report on "Jada and Will Smith's marital status" regarding the year they began their separation?
  retrieval: found 1/2 of the documents the answer needs, 15 returned
  recall@5: 0.50   precision@5: 0.20   time: 2464ms
  expander.queries:
    Did The Independent Life and Style article about Jada Pinkett Smith and Will Smith's marriage conflict with ...
    Was there a discrepancy in The Independent - Life and Style between the article on Jada Pinkett Smith and Wi...
    Did The Independent publish contradictory information in its Life and Style pieces on Jada and Will Smith re...
    Did "The Independent - Life and Style" article on "Jada Pinkett Smith and Will Smith's marriage" disagree wi...
  needed but never retrieved: [doc db0a9f09] Outsiders are now aware that Jada and Will have been separated since 2016, but the timeline of h...

Testing

New tests were added and the new example script was run locally

Notes for reviewers

The tracing package is included because the evaluator depends on it for collecting per evaluation case component data.

Checklist

@sjrl sjrl self-assigned this Sep 11, 2026
@github-actions github-actions Bot added the type:documentation Improvements or additions to documentation label Sep 11, 2026
@github-actions

github-actions Bot commented Sep 11, 2026 •

Copy link
Copy Markdown
Contributor

Coverage report (agent_pack)

Click to see where and how coverage changed

FileStatementsMissingCoverageCoverage
(new stmts)
Lines missing
  integrations/agent_pack/src/haystack_integrations/agent_pack/evaluation
  dataclasses.py
  harness_log_collector.py 74-75, 107
  retrieval_harness_evaluator.py 153, 314
  tracer.py
Project Total  

This report was generated by python-coverage-comment-action

@sjrl
sjrl marked this pull request as ready for review September 11, 2026 11:33
@sjrl
sjrl requested a review from a team as a code owner September 11, 2026 11:33
@sjrl
sjrl requested review from anakin87 and julian-risch and removed request for a team September 11, 2026 11:33
@sjrl

sjrl commented Sep 11, 2026

Copy link
Copy Markdown
Contributor Author

@julian-risch would also appreciate your review due to your experience in evaluation and that we need this for a talk submission.

@anakin87

Copy link
Copy Markdown
Member

In general, understanding a bit about how you imagine this Agent overall and listing the pieces needed to achieve the goal would ease review.

@sjrl

sjrl commented Sep 14, 2026

Copy link
Copy Markdown
Contributor Author

In general, understanding a bit about how you imagine this Agent overall and listing the pieces needed to achieve the goal would ease review.

Sounds good! If you want to get a better idea of how it all works together I recommend looking at #3932 and this example file in particular.

I can work on a conceptual write up later today.

@sjrl

sjrl commented Sep 14, 2026

Copy link
Copy Markdown
Contributor Author

In general, understanding a bit about how you imagine this Agent overall and listing the pieces needed to achieve the goal would ease review.

@anakin87 and @julian-risch I've provided a high-level view of what I'm trying to accomplish in this comment https://github.com/deepset-ai/haystack-private/issues/498#issuecomment-5663552339 and how the evaluation harnesses fit into this.

@julian-risch julian-risch left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Other than restoring a previously installed tracer and extending the validation, I don't have any other comments. Let's not block the PR because of that. I see Stefano had some good comments.

By the way, running some experiments, my agent frequently came up with pipelines that had errors, for example wrong socket names.


def validate(self, target: Pipeline) -> None:
"""
Validate the candidate pipeline to expose at least one `query` input and exactly one `documents` output.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This simple validation is good as a start. Checking that there are also no mandatory unconnected inputs other than query should be the next step. Using is_mandatory.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good idea, I'll note this down to follow up on, but not include in this PR

@sjrl

sjrl commented Sep 25, 2026

Copy link
Copy Markdown
Contributor Author

By the way, running some experiments, my agent frequently came up with pipelines that had errors, for example wrong socket names.

@julian-risch what model were you using for making the pipeline? And is this something that you think would be fixed by extending the validation as you suggested?

@julian-risch

Copy link
Copy Markdown
Member

By the way, running some experiments, my agent frequently came up with pipelines that had errors, for example wrong socket names.

@julian-risch what model were you using for making the pipeline? And is this something that you think would be fixed by extending the validation as you suggested?

Relatively small models. Nemotron-3.5-Lightning-30B-A3B

I don't think extending the validation will fix the particular challenges I faced, no. Just showing that the validation step itself is important to guide the agent to come up with runnable pipelines.
Some example errors in case you're interested:

  • "PipelineConnectError: 'retriever.documents does not exist. Input connections of retriever are: queries (type list[str]), retriever_kwargs (type dict[str, Any] | None)"
  • "PipelineConnectError: 'expander.query' does not exist. Output connections of expander are: queries (type list[str])"
  • 'ValueError: Component named documents not found in the pipeline.'
  • 'ValueError: Component named rankers not found in the pipeline.'}

@sjrl
sjrl requested a review from anakin87 September 25, 2026 11:40
@sjrl
sjrl merged commit 04fc30b into main Sep 25, 2026
19 checks passed
@sjrl
sjrl deleted the agent-pack/retrieval-harness-evaluator branch September 25, 2026 11:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

integration:agent-pack type:documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants