Evaluate source-grounded agents on the quality of their evidence and the usefulness of their recommendations.
- Did the agent find fresh, relevant sources?
- Did extraction return enough useful content?
- Are claims supported by source evidence?
- Are recommendations specific and actionable?
- Did the workflow stay within cost and latency budgets?
- Can a user inspect where the answer came from?
| Dimension | What To Check |
|---|---|
| Freshness | Sources reflect the requested time window |
| Relevance | Results match the approved context |
| Coverage | Prospect, industry, and competitor lanes are represented |
| Grounding | Every important claim maps to a source |
| Specificity | Recommendations name concrete actions |
| Source diversity | The brief does not depend on one domain |
| Cost | Run stays within search, extraction, and token budgets |
| Latency | User sees progress and receives output fast enough |
| Robustness | Partial failures do not break the whole run |
Keep a small fixture set for repeatable tests:
- a known company profile
- expected competitors
- representative search results
- extracted page samples
- one high-quality target brief
- examples of weak or irrelevant sources
Use the fixture set to test prompt changes, schema edits, and UI changes without spending provider credits.
For every generated brief, validate:
- cited URLs appear in the evidence packet
- source titles are preserved
- no final section cites an invented source
- unsupported claims are either removed or marked uncertain
For strategy-style outputs, a human reviewer should ask:
- Would I act on this today?
- Is the recommendation grounded in current evidence?
- Is the source credible enough for the action?
- Did the agent miss an obvious source category?
- Is the brief concise enough for the target workflow?