[sjrl]

Related Issues

• part of deepset-ai/haystack-private#498

Proposed Changes:

Second part of the harness optimization work: the loop that runs the optimizer agent from #4037 over a labelled evaluation set.

• HarnessOptimizationExperiment(reference, evaluator, eval_cases, prices, objectives, journal, optimizer_*, ...) builds its own optimizer agent from its evaluator and the loader that matches the reference.
• run() measures the reference, then for up to max_iterations turns proposes a candidate, measures, prices and gates it. Each turn starts from the best candidate so far.
• Returns an ExperimentResult: every CandidateEvaluation, the gate failures, the optimizer's own usage and cost, and an ExperimentRecommendation when a candidate beats the reference.
• Every measurement is appended to an ExperimentJournal, and each run writes its YAML artifacts to its own directory. on_baseline and on_candidate report progress.
• Example: retrieval_pipeline_optimization.py runs a full experiment on a retrieval pipeline.

How did you test it?

Unit tests in tests/optimization/test_experiment.py with scripted optimizer models. hatch run test:unit, hatch run test:types and hatch run fmt-check pass locally.

Ran the new example end to end on the MultiHopRAG corpus (needs OPENAI_API_KEY):

hatch run test:python examples/retrieval_pipeline_optimization.py

I'd recommend starting the review with examples/retrieval_pipeline_optimization.py. It shows how a reference pipeline, an evaluator, eval cases and objectives come together in HarnessOptimizationExperiment, and prints each candidate as it is measured.

Notes for the reviewer

• Based on main, which now has the optimizer agent from feat: Add harness optimizer agent and its configuration editing tools #4037.
• The RAG agent harness evaluator follows in feat: Add Advanced RAG harness evaluator #3932, stacked on this PR.

Checklist

• I have read the contributors guidelines and the code of conduct
• I have updated the related issue with new insights and changes
• I added unit tests and updated the docstrings
• I've used one of the conventional commit types for my PR title: fix:, feat:, build:, chore:, ci:, docs:, style:, refactor:, perf:, test:.

[github-actions]

Coverage report (agent_pack)

Click to see where and how coverage changed

File	Statements	Missing	Coverage	Coverage (new stmts)	Lines missing
integrations/agent_pack/src/haystack_integrations/agent_pack/optimization
agent.py
dataclasses.py
experiment.py					83-85, 185-186, 237-238, 242, 336, 527, 535
tools.py
Project Total

This report was generated by python-coverage-comment-action

[sjrl]

@julian-risch sorry meant to be a draft PR