Repository navigation
Demo: record_step() vs the manual Step-building pattern - #6
Open
jasmine-ab-tea wants to merge 4 commits into
Open
jasmine-ab-tea wants to merge 4 commits into
jasmine-ab-tea wants to merge 4 commits into
Conversation
agent-control PR #287 adds agent_control.record_step()/StepRecorder as an incremental way to build trace/session Step trees. This script builds the same trace, session, and empty-children scenarios both the new way and the existing hand-rolled StepExecution + dict-record + build_agent_control_step() way already used by banking_multilevel_cases.py, asserts the two produce equivalent Steps, and prints the source side by side so the ergonomics difference is visible directly in the code. Needs agent-control-sdk>=8.11.0 (unreleased as of this commit) for the record_step()/evaluate_step() path; the manual path runs on the currently-published SDK.
record_step()/.child() both accept a context= kwarg, so the new-way builders can carry the same per-node metadata the old way's record dicts do. The two tool leaves built via .call() (account lookup, submit_transfer) switch to .child() instead, since .call()'s auto-capture never sets a context key. The equivalence check now compares full Step.model_dump() output directly instead of stripping context first. Co-Authored-By: Claude Code <noreply@anthropic.com>
With StepRecorder.call()/acall() now accepting step_context= (agent-control yixint/pr271-followup-children-ergonomics @ 93dfa4d), the account-lookup leaf in new_way_trace can go back to .call() instead of .child(): its auto-derived input already matches the old way's tool_input dict, and step_context carries the per-leaf metadata without colliding with a same-named kwarg on the wrapped function. The retriever and llm leaves stay on .child() - .call()'s auto-derived input wouldn't match their desired shape (bare query string / chat-message list). submit_transfer also stays on .child(): its metadata depends on sandbox state only known after the call runs, but step_context is evaluated eagerly before the call happens. Co-Authored-By: Claude Code <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
step_recorder_vs_manual_demo.py, a standalone script that builds three scenarios - a trace, a session nesting two traces, and a trace that ran with zero children - two ways:agent_control.record_step()/StepRecorder(.child(),.call(),.build(),.evaluate()).StepExecution+ dict-record pattern (banking_multilevel_cases.py'strace_record/session_record, converted viabanking_multilevel_streamlit_app.py'sbuild_agent_control_step()).Steptrees are equivalent (same type/name/input/output/tools/children;contextbookkeeping metadata is intentionally excluded from the comparison since it's caller-defined and doesn't affect evaluation) and prints both sides' source so the ergonomics difference - noStepExecutionwrapper, no dict record, no recursive converter - is visible directly in the code, not just claimed in prose.record_step(..., container_types={"trace"}), which getschildren=[]instead of omitted with one keyword, versus the old way's "remember to add"spans": []by hand" equivalent.--mode {old,new,both}and--scenario {trace,session,empty,all}let a reader run either path independently.--evaluateadditionally sends both built Steps to a live Agent Control + Galileo devstack (same env vars as the rest of this repo'sAGENTS.md);GALILEO_API_KEYis only ever read from the environment, never printed.Verified locally
Editable-installed the agent-control PR #287 branch's
models/engine/telemetry/sdk/evaluatorspackages into this repo's local.venv(not committed - the publishedagent-control-sdkon the registry is still 8.7.0, which predatesrecord_step()/Step.children) and ran every scenario in--mode both: all three reportEQUIVALENT. Also ran--evaluateagainst the real devstack end to end with no errors.Note for reviewers: until
agent-control-sdk>=8.11.0is published,--mode new/--mode bothwon't importrecord_stepfrom a freshly-uv sync'd environment.--mode oldruns today against the currently published SDK.Test plan
python step_recorder_vs_manual_demo.py(needsagent-control-sdk>=8.11.0) - all three scenarios printEQUIVALENTpython step_recorder_vs_manual_demo.py --mode old- runs against today's published SDK with no errorspython step_recorder_vs_manual_demo.py --scenario trace --evaluatewith devstack env vars set - prints match counts for both ways, no credential values printed🤖 Generated with Claude Code