Repository navigation
feat(strands): trace decision models as decision spans - #1090
Open
Abhijeet Prasad (AbhiPrasad) wants to merge 1 commit into
Open
Abhijeet Prasad (AbhiPrasad) wants to merge 1 commit into
Abhijeet Prasad (AbhiPrasad) wants to merge 1 commit into
Conversation
> 🤖 **This text was AI-generated.**
strands-agents 1.63 added `DecisionModel`, and 1.64 added two hosted
implementations, `DecisionsAPIDecisionModel` and `TypeSafeDecisionModel`.
None of them go through `strands.telemetry.Tracer`, which is the only thing
the Strands integration patched, so decision calls produced no Braintrust
spans:
```python
result = await TypeSafeDecisionModel().ask(
"I was charged twice, please refund one charge.",
{"department": {"question": "Which team?", "choices": ["billing", "support"]}},
)
# before: no span
```
The integration now patches `DecisionModel.ask`. Every implementation
inherits it, so the one patch covers `LLMDecisionModel`, `TypeSafeDecisionModel`
and `DecisionsAPIDecisionModel`. Each call becomes a `decision` span that
follows the decision-spans spec:
https://github.com/braintrustdata/braintrust-spec/blob/main/skills/instrumentation-spec/references/features/decision-spans.md
```
bt-agent.invoke
└── event_loop.cycle
└── route_ticket.execute (tool)
├── LLMDecisionModel.ask (decision)
│ └── <provider LLM span, if that integration is on>
└── TypeSafeDecisionModel.ask (decision)
```
- input: `{"state": ..., "questions": [{"id", "type", "instructions", "criteria"}]}`.
Yes/no questions are `boolean`; choice lists become `choice` with criteria.
- output: `{"answers": [...]}`, linked to questions by `id`. Boolean answers
carry `probability`; choice answers carry `choice` plus the provider's
`probabilities` and `confidence` when it returns them. `LLMDecisionModel`
forces a true/false answer, recorded as probability 1.0 or 0.0. When
strands returns `Uncertain` because a System One answer fell below its
confidence cutoff, the provider's actual answer is logged; when the model
itself declined, the answer is a `refusal`.
- metadata: `provider`, `model` (the provider-reported model, e.g.
`jev-1.14.0` for a requested `jev-latest`), `decision_model`, `latency_ms`.
- metrics: TypeSafe and DecisionsAPI calls go over raw HTTP, so their token
usage is recorded as standard token metrics. `LLMDecisionModel` keeps usage
in `metadata.strands_usage`, because the wrapped model's own provider
integration already records those tokens on a child LLM span.
- attachments: inline image/document/video bytes in `state` are logged as
Braintrust attachments in place; the request sent to the model is unchanged.
Decision spans nest under the active Strands span, for example the tool that
called `ask()`. Without an OTEL SDK, Strands' spans are invalid and the
integration stands in a key object that `get_current_span()` rejects, so the
parent is read from the raw OTEL context value
(`opentelemetry.trace.propagation._SPAN_KEY`, a private name, guarded).
Instrumentation failures are caught and never break the `ask()` call.
Limitations: users running an OTEL SDK with Braintrust's OTEL processor will
also see Strands' own OTEL `decision` span. Inline media is only converted to
attachments on decision spans, not on the other Strands spans.
Closes #1087
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
Luca Forstner (lforst)
approved these changes
Nov 9, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
strands-agents 1.63 added
DecisionModel, and 1.64 added two hosted implementations,DecisionsAPIDecisionModelandTypeSafeDecisionModel. None of them go throughstrands.telemetry.Tracer, which is the only thing the Strands integration patched, so decision calls produced no Braintrust spans:The integration now patches
DecisionModel.ask. Every implementation inherits it, so the one patch coversLLMDecisionModel,TypeSafeDecisionModelandDecisionsAPIDecisionModel. Each call becomes adecisionspan that follows the decision-spans spec:https://github.com/braintrustdata/braintrust-spec/blob/main/skills/instrumentation-spec/references/features/decision-spans.md
{"state": ..., "questions": [{"id", "type", "instructions", "criteria"}]}. Yes/no questions areboolean; choice lists becomechoicewith criteria.{"answers": [...]}, linked to questions byid. Boolean answers carryprobability; choice answers carrychoiceplus the provider'sprobabilitiesandconfidencewhen it returns them.LLMDecisionModelforces a true/false answer, recorded as probability 1.0 or 0.0. When strands returnsUncertainbecause a System One answer fell below its confidence cutoff, the provider's actual answer is logged; when the model itself declined, the answer is arefusal.provider,model(the provider-reported model, e.g.jev-1.14.0for a requestedjev-latest),decision_model,latency_ms.LLMDecisionModelkeeps usage inmetadata.strands_usage, because the wrapped model's own provider integration already records those tokens on a child LLM span.stateare logged as Braintrust attachments in place; the request sent to the model is unchanged.Decision spans nest under the active Strands span, for example the tool that called
ask(). Without an OTEL SDK, Strands' spans are invalid and the integration stands in a key object thatget_current_span()rejects, so the parent is read from the raw OTEL context value(
opentelemetry.trace.propagation._SPAN_KEY, a private name, guarded). Instrumentation failures are caught and never break theask()call.Limitations: users running an OTEL SDK with Braintrust's OTEL processor will also see Strands' own OTEL
decisionspan. Inline media is only converted to attachments on decision spans, not on the other Strands spans.Closes #1087