Skip to content

feat(strands): trace decision models as decision spans - #1090

Open
Abhijeet Prasad (AbhiPrasad) wants to merge 1 commit into
mainfrom
abhi/strands-decision-tracing-ac2cc366
Open

Abhijeet Prasad (AbhiPrasad) wants to merge 1 commit into
mainfrom
abhi/strands-decision-tracing-ac2cc366

Conversation

@AbhiPrasad

Copy link
Copy Markdown
Member

🤖 This text was AI-generated.

strands-agents 1.63 added DecisionModel, and 1.64 added two hosted implementations, DecisionsAPIDecisionModel and TypeSafeDecisionModel. None of them go through strands.telemetry.Tracer, which is the only thing the Strands integration patched, so decision calls produced no Braintrust spans:

result = await TypeSafeDecisionModel().ask(
    "I was charged twice, please refund one charge.",
    {"department": {"question": "Which team?", "choices": ["billing", "support"]}},
)
# before: no span

The integration now patches DecisionModel.ask. Every implementation inherits it, so the one patch covers LLMDecisionModel, TypeSafeDecisionModel and DecisionsAPIDecisionModel. Each call becomes a decision span that follows the decision-spans spec:
https://github.com/braintrustdata/braintrust-spec/blob/main/skills/instrumentation-spec/references/features/decision-spans.md

bt-agent.invoke
└── event_loop.cycle
    └── route_ticket.execute                (tool)
        ├── LLMDecisionModel.ask            (decision)
        │   └── <provider LLM span, if that integration is on>
        └── TypeSafeDecisionModel.ask       (decision)
  • input: {"state": ..., "questions": [{"id", "type", "instructions", "criteria"}]}. Yes/no questions are boolean; choice lists become choice with criteria.
  • output: {"answers": [...]}, linked to questions by id. Boolean answers carry probability; choice answers carry choice plus the provider's probabilities and confidence when it returns them. LLMDecisionModel forces a true/false answer, recorded as probability 1.0 or 0.0. When strands returns Uncertain because a System One answer fell below its confidence cutoff, the provider's actual answer is logged; when the model itself declined, the answer is a refusal.
  • metadata: provider, model (the provider-reported model, e.g. jev-1.14.0 for a requested jev-latest), decision_model, latency_ms.
  • metrics: TypeSafe and DecisionsAPI calls go over raw HTTP, so their token usage is recorded as standard token metrics. LLMDecisionModel keeps usage in metadata.strands_usage, because the wrapped model's own provider integration already records those tokens on a child LLM span.
  • attachments: inline image/document/video bytes in state are logged as Braintrust attachments in place; the request sent to the model is unchanged.

Decision spans nest under the active Strands span, for example the tool that called ask(). Without an OTEL SDK, Strands' spans are invalid and the integration stands in a key object that get_current_span() rejects, so the parent is read from the raw OTEL context value
(opentelemetry.trace.propagation._SPAN_KEY, a private name, guarded). Instrumentation failures are caught and never break the ask() call.

Limitations: users running an OTEL SDK with Braintrust's OTEL processor will also see Strands' own OTEL decision span. Inline media is only converted to attachments on decision spans, not on the other Strands spans.

Closes #1087

> 🤖 **This text was AI-generated.**

strands-agents 1.63 added `DecisionModel`, and 1.64 added two hosted
implementations, `DecisionsAPIDecisionModel` and `TypeSafeDecisionModel`.
None of them go through `strands.telemetry.Tracer`, which is the only thing
the Strands integration patched, so decision calls produced no Braintrust
spans:

```python
result = await TypeSafeDecisionModel().ask(
    "I was charged twice, please refund one charge.",
    {"department": {"question": "Which team?", "choices": ["billing", "support"]}},
)
# before: no span
```

The integration now patches `DecisionModel.ask`. Every implementation
inherits it, so the one patch covers `LLMDecisionModel`, `TypeSafeDecisionModel`
and `DecisionsAPIDecisionModel`. Each call becomes a `decision` span that
follows the decision-spans spec:
https://github.com/braintrustdata/braintrust-spec/blob/main/skills/instrumentation-spec/references/features/decision-spans.md

```
bt-agent.invoke
└── event_loop.cycle
    └── route_ticket.execute                (tool)
        ├── LLMDecisionModel.ask            (decision)
        │   └── <provider LLM span, if that integration is on>
        └── TypeSafeDecisionModel.ask       (decision)
```

- input: `{"state": ..., "questions": [{"id", "type", "instructions", "criteria"}]}`.
  Yes/no questions are `boolean`; choice lists become `choice` with criteria.
- output: `{"answers": [...]}`, linked to questions by `id`. Boolean answers
  carry `probability`; choice answers carry `choice` plus the provider's
  `probabilities` and `confidence` when it returns them. `LLMDecisionModel`
  forces a true/false answer, recorded as probability 1.0 or 0.0. When
  strands returns `Uncertain` because a System One answer fell below its
  confidence cutoff, the provider's actual answer is logged; when the model
  itself declined, the answer is a `refusal`.
- metadata: `provider`, `model` (the provider-reported model, e.g.
  `jev-1.14.0` for a requested `jev-latest`), `decision_model`, `latency_ms`.
- metrics: TypeSafe and DecisionsAPI calls go over raw HTTP, so their token
  usage is recorded as standard token metrics. `LLMDecisionModel` keeps usage
  in `metadata.strands_usage`, because the wrapped model's own provider
  integration already records those tokens on a child LLM span.
- attachments: inline image/document/video bytes in `state` are logged as
  Braintrust attachments in place; the request sent to the model is unchanged.

Decision spans nest under the active Strands span, for example the tool that
called `ask()`. Without an OTEL SDK, Strands' spans are invalid and the
integration stands in a key object that `get_current_span()` rejects, so the
parent is read from the raw OTEL context value
(`opentelemetry.trace.propagation._SPAN_KEY`, a private name, guarded).
Instrumentation failures are caught and never break the `ask()` call.

Limitations: users running an OTEL SDK with Braintrust's OTEL processor will
also see Strands' own OTEL `decision` span. Inline media is only converted to
attachments on decision spans, not on the other Strands spans.

Closes #1087
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 9, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-09T20:08:09.785697Z fb656d3 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

strands 1.64: trace new decision models

2 participants