Skip to content

test: harder routing evals, 7931 canary eval - #8

Merged
dd-Splunk merged 1 commit into
mainfrom
test/evals-harder
Oct 28, 2026
Merged

dd-Splunk merged 1 commit into
mainfrom
test/evals-harder

Conversation

@dd-Splunk

Copy link
Copy Markdown
Owner

Summary

  • Add near-miss routing evals: upgrade-safety advice (negative), Splunk Operator for Kubernetes (negative), compose help with no Splunkbase app (negative), version pick for a target Splunk (positive), name-only compose request (positive).
  • Add canary_image_version, a behavior eval on app 7931 that checks the Splunk version is taken from an image tag and compatibility comes from script output, not memory.
  • Fix a literal \n in the outdated_report_not_advice grading instructions.
  • Evals only: no skill or script change, so no version bump or release. Total is now 20 evals.

Test plan

  • Unit tests pass (149, 8 live skipped) and ruff clean
  • The 5 new routing prompts and the canary behavior eval were run through subagents and matched their expected outcome
  • CI green

🤖 Generated with Claude Code

…xt escape

Co-Authored-By: Claude Code <noreply@anthropic.com>
@dd-Splunk
dd-Splunk merged commit 960d8c8 into main Oct 28, 2026
6 checks passed
@dd-Splunk
dd-Splunk deleted the test/evals-harder branch October 28, 2026 14:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant