Repository navigation
feat: Add harness evaluator for retrieval pipeline - #3947
Conversation
Coverage report (agent_pack)Click to see where and how coverage changed
This report was generated by python-coverage-comment-action |
||||||||||||||||||||||||||||||||||||||||||
|
@julian-risch would also appreciate your review due to your experience in evaluation and that we need this for a talk submission. |
|
In general, understanding a bit about how you imagine this Agent overall and listing the pieces needed to achieve the goal would ease review. |
Sounds good! If you want to get a better idea of how it all works together I recommend looking at #3932 and this example file in particular. I can work on a conceptual write up later today. |
@anakin87 and @julian-risch I've provided a high-level view of what I'm trying to accomplish in this comment https://github.com/deepset-ai/haystack-private/issues/498#issuecomment-5663552339 and how the evaluation harnesses fit into this. |
julian-risch
left a comment
There was a problem hiding this comment.
Other than restoring a previously installed tracer and extending the validation, I don't have any other comments. Let's not block the PR because of that. I see Stefano had some good comments.
By the way, running some experiments, my agent frequently came up with pipelines that had errors, for example wrong socket names.
|
|
||
| def validate(self, target: Pipeline) -> None: | ||
| """ | ||
| Validate the candidate pipeline to expose at least one `query` input and exactly one `documents` output. |
There was a problem hiding this comment.
This simple validation is good as a start. Checking that there are also no mandatory unconnected inputs other than query should be the next step. Using is_mandatory.
There was a problem hiding this comment.
Good idea, I'll note this down to follow up on, but not include in this PR
@julian-risch what model were you using for making the pipeline? And is this something that you think would be fixed by extending the validation as you suggested? |
Relatively small models. Nemotron-3.5-Lightning-30B-A3B I don't think extending the validation will fix the particular challenges I faced, no. Just showing that the validation step itself is important to guide the agent to come up with runnable pipelines.
|
Related Issues
Proposed Changes:
This PR is another reviewable slice of the larger harness evaluator/optimization work in #3932.
It introduces
HarnessEvaluator, which evaluates an Agent or Pipeline against a set of eval cases and returns normalizedEvaluationMetrics, including quality, latency, per-model token usage, and evaluator-specific details.The first implementation,
RetrievalHarnessEvaluator, evaluates retrieval pipelines by discovering query inputs and a single documents output from socket names rather than component names. This lets the same evaluator work with anything from a simple BM25 retriever to a multi-stage retrieval/reranking pipeline.validate()rejects unsupported pipelines before evaluation starts.Each
RetrievalEvalCasecontains a question and the expected evidence by document ID. Cases are scored withrecall@k,precision@k, andfound_at. Overall quality is based on recall@k, so partial retrieval improvements are reflected in the score.Supporting pieces included in this PR:
RetrievalEvalCaseand retrieval scoring utilities.haystack_integrations.tracing.agent_pack, a lightweight tracer used to collect per-model token usage, per-component output sizes, and small socket samples.ComponentLogCollectorfor capturing warnings emitted by components.examples/retrieval_eval.py, which evaluates a BM25 pipeline over the MultiHopRAG corpus and reports aggregate and per evaluation case results.Here is what running the new
retrieval_eval.pyexample file producesTesting
New tests were added and the new example script was run locally
Notes for reviewers
The tracing package is included because the evaluator depends on it for collecting per evaluation case component data.
Checklist
fix:,feat:,build:,chore:,ci:,docs:,style:,refactor:,perf:,test:.