[opentelemetry-instrumentation-genai-anthropic] Instrument beta.messages create, stream, and parse (#550) by benkawecki · Pull Request #781 · open-telemetry/opentelemetry-python-genai

benkawecki · 2026-09-23T20:06:07Z

Description

Fixes 550

This work builds from PR 661

Instruments anthropic.resources.beta.messages.Messages and AsyncMessages (create, stream, and parse), including raw and streaming response helpers.

Anthropic beta messages provide access to new API features (such as prompt caching, computer use, and MCP tools) while producing standard chat inference spans and metrics.

Type of change

New feature (non-breaking change which adds functionality)

How has this been tested?

Please describe the tests that you ran to verify your changes. Provide
instructions so we can reproduce. List any relevant details for your test
configuration.

uv run tox -e py312-test-instrumentation-genai-anthropic-conformance
uv run tox -e py312-test-instrumentation-genai-anthropic-latest
uv run tox -e py312-test-instrumentation-genai-anthropic-oldest

Checklist

See CONTRIBUTING.md
for the style guide, changelog guidance, and more.

Followed the style guidelines of this project
Changelog updated if the change requires an entry
Unit tests added
Documentation updated

opentelemetry-pr-dashboard · 2026-09-23T20:07:59Z

Pull request dashboard status

Waiting on the author · refreshed 2026-10-17 01:44 UTC

Respond to 1 review item (e.g. link a commit, explain why not, ask a follow-up):

Top-level threads: 1

Status above doesn't look right?

Just replied or pushed? Anything around or after the refresh time above may not be picked up yet — give it a few minutes.
Should this be with reviewers? Comment /dashboard route:reviewers to route it to them.
Anything wrong — including the routing? Report it with what you expected; it helps us improve the dashboard.

Copilot

Copilot review overview

🟡 Changes recommended

Redact exposed workspace identifiers and correct the beta raw-response coverage and duplicate conformance registration.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 5 · 1

Open (6)

Redact anthropic-workspace-id from cassette · New
Redact anthropic-workspace-id from cassette · New
Redact anthropic-workspace-id from cassette · New
Redact anthropic-workspace-id from cassette · New
Use beta resource for beta raw-response streaming test · New
Remove duplicate InferenceBetaStreamingScenario entry · New

What changed in this PR

Adds instrumentation for Anthropic beta Messages APIs, including synchronous/asynchronous create, stream, parse, raw responses, and content extraction.

Changes:

Adds beta-aware wrappers, parsing, streaming, and telemetry extraction.
Adds unit, conformance, and cassette-backed tests.
Updates documentation and changelog.

File	Summary
`instrumentation/opentelemetry-instrumentation-genai-anthropic/tests/test_sync_messages.py`	Tests synchronous beta messages.
`instrumentation/opentelemetry-instrumentation-genai-anthropic/tests/test_parse_messages.py`	Tests beta parse behavior.
`instrumentation/opentelemetry-instrumentation-genai-anthropic/tests/test_messages_extractors.py`	Tests beta content extraction.
`instrumentation/opentelemetry-instrumentation-genai-anthropic/tests/test_instrumentor.py`	Tests instrumentation lifecycle.
`instrumentation/opentelemetry-instrumentation-genai-anthropic/tests/test_conformance.py`	Registers conformance scenarios.
`instrumentation/opentelemetry-instrumentation-genai-anthropic/tests/test_async_wrappers.py`	Tests asynchronous stream handling.
`instrumentation/opentelemetry-instrumentation-genai-anthropic/tests/test_async_messages.py`	Tests asynchronous beta messages.
`instrumentation/opentelemetry-instrumentation-genai-anthropic/tests/conformance/tool_calling_beta.py`	Adds beta tool-calling coverage.
`instrumentation/opentelemetry-instrumentation-genai-anthropic/tests/conformance/inference_streaming_beta.py`	Adds beta streaming coverage.
`instrumentation/opentelemetry-instrumentation-genai-anthropic/tests/conformance/inference_raw_response_beta.py`	Adds beta raw-response coverage.
`instrumentation/opentelemetry-instrumentation-genai-anthropic/tests/conformance/inference_beta.py`	Adds beta inference coverage.
`instrumentation/opentelemetry-instrumentation-genai-anthropic/tests/conformance/inference_beta_server_tool_calling.py`	Adds beta server tool-calling coverage.
`instrumentation/opentelemetry-instrumentation-genai-anthropic/tests/cassettes/tool_calling_beta_conformance.yaml`	Records beta tool-calling interactions.
`instrumentation/opentelemetry-instrumentation-genai-anthropic/tests/cassettes/test_sync_messages_beta_create_with_raw_response.yaml`	Records synchronous beta raw responses.
`instrumentation/opentelemetry-instrumentation-genai-anthropic/tests/cassettes/test_sync_messages_beta_create_streaming.yaml`	Records synchronous beta streaming.
`instrumentation/opentelemetry-instrumentation-genai-anthropic/tests/cassettes/test_sync_messages_beta_create_streaming_with_raw_response.yaml`	Records synchronous beta raw streaming.
`instrumentation/opentelemetry-instrumentation-genai-anthropic/tests/cassettes/test_sync_beta_messages_with_raw_response.yaml`	Records synchronous beta raw responses.
`instrumentation/opentelemetry-instrumentation-genai-anthropic/tests/cassettes/test_sync_beta_messages_stream.yaml`	Records synchronous beta streams.
`instrumentation/opentelemetry-instrumentation-genai-anthropic/tests/cassettes/test_sync_beta_messages_parse_basic.yaml`	Records synchronous beta parsing.
`instrumentation/opentelemetry-instrumentation-genai-anthropic/tests/cassettes/test_sync_beta_messages_create_with_content.yaml`	Records beta content creation.
`instrumentation/opentelemetry-instrumentation-genai-anthropic/tests/cassettes/test_sync_beta_messages_create_streaming.yaml`	Records synchronous beta streaming.
`instrumentation/opentelemetry-instrumentation-genai-anthropic/tests/cassettes/test_sync_beta_messages_create_basic.yaml`	Records basic synchronous beta creation.
`instrumentation/opentelemetry-instrumentation-genai-anthropic/tests/cassettes/test_sync_beta_messages_create_api_error.yaml`	Records synchronous beta API errors.
`instrumentation/opentelemetry-instrumentation-genai-anthropic/tests/cassettes/test_async_beta_messages_with_raw_response.yaml`	Records asynchronous beta raw responses.
`instrumentation/opentelemetry-instrumentation-genai-anthropic/tests/cassettes/test_async_beta_messages_stream.yaml`	Records asynchronous beta streams.
`instrumentation/opentelemetry-instrumentation-genai-anthropic/tests/cassettes/test_async_beta_messages_parse_basic.yaml`	Records asynchronous beta parsing.
`instrumentation/opentelemetry-instrumentation-genai-anthropic/tests/cassettes/test_async_beta_messages_create_with_content.yaml`	Records asynchronous beta content creation.
`instrumentation/opentelemetry-instrumentation-genai-anthropic/tests/cassettes/test_async_beta_messages_create_streaming.yaml`	Records asynchronous beta streaming.
`instrumentation/opentelemetry-instrumentation-genai-anthropic/tests/cassettes/test_async_beta_messages_create_basic.yaml`	Records basic asynchronous beta creation.
`instrumentation/opentelemetry-instrumentation-genai-anthropic/tests/cassettes/test_async_beta_messages_create_api_error.yaml`	Records asynchronous beta API errors.
`instrumentation/opentelemetry-instrumentation-genai-anthropic/tests/cassettes/inference_beta_streaming_conformance.yaml`	Records beta streaming conformance.
`instrumentation/opentelemetry-instrumentation-genai-anthropic/tests/cassettes/inference_beta_server_tool_calling_conformance.yaml`	Records beta server tool-calling conformance.
`instrumentation/opentelemetry-instrumentation-genai-anthropic/tests/cassettes/inference_beta_conformance.yaml`	Records beta inference conformance.
`instrumentation/opentelemetry-instrumentation-genai-anthropic/src/opentelemetry/instrumentation/genai/anthropic/wrappers.py`	Handles beta stream accumulation.
`instrumentation/opentelemetry-instrumentation-genai-anthropic/src/opentelemetry/instrumentation/genai/anthropic/utils.py`	Converts beta content blocks.
`instrumentation/opentelemetry-instrumentation-genai-anthropic/src/opentelemetry/instrumentation/genai/anthropic/patch.py`	Instruments beta calls and streams.
`instrumentation/opentelemetry-instrumentation-genai-anthropic/src/opentelemetry/instrumentation/genai/anthropic/messages_extractors.py`	Extracts beta request and response data.
`instrumentation/opentelemetry-instrumentation-genai-anthropic/src/opentelemetry/instrumentation/genai/anthropic/_raw_response.py`	Supports beta raw responses.
`instrumentation/opentelemetry-instrumentation-genai-anthropic/src/opentelemetry/instrumentation/genai/anthropic/__init__.py`	Registers beta instrumentation.
`instrumentation/opentelemetry-instrumentation-genai-anthropic/README.rst`	Documents beta API support.
`instrumentation/opentelemetry-instrumentation-genai-anthropic/.changelog/661.added`	Adds the feature changelog entry.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

benkawecki · 2026-09-23T20:12:02Z

A couple notes / questions for maintainers.

I took over @lmolkova PR 661 code for this. There were some hard conflicts between that branch and main. I ended up squashing the existing commits and applying those on top of main. This does however remove the commit messages and contribute history from @lmolkova which I don't want to do unless thats okay with others. My intent wasn't to claim that work, but rather the to avoid a really nasty rebase. If people think preserving that is more important I can go back, do the fully rebase and then cherry pick my work ontop.
I found a failing test when running uv run tox -e py314-test-instrumentation-genai-anthropic-oldest. It appears this fails on the current main branch as well. Please let me know if I should open an issue or attempt to resolve the issue in this PR.
The current anthropic model for the conformance tests (claude-sonnet-4-20250514) is deprecated and no longer works. I updated to sonnet 4.6 for my tests. I can updating the existing tests and re-run the VCR capture in this PR if people think thats prudent.

aabmass · 2026-09-23T21:19:04Z

This does however remove the commit messages and contribute history from @lmolkova which I don't want to do unless thats okay with others. My intent wasn't to claim that work, but rather the to avoid a really nasty rebase.

She's OOO but I don't think it's a problem. Can you add her email as a coauthor on https://docs.github.com/en/pull-requests/how-tos/commit-changes/creating-a-commit-with-multiple-authors?

2. It appears this fails on the current main branch as well. Please let me know if I should open an issue or attempt to resolve the issue in this PR.

If it's a consistent failure, sending a separate PR would be awesome :)

3. I can updating the existing tests and re-run the VCR capture in this PR if people think thats prudent.

My 2c, that would be great but consider doing in a separate PR. Thanks!

…sages create, stream, and parse Adds the beta.messages instrumentation (create/stream/parse spans, raw-response support, server-tool-calling and MCP content classification, conformance scenarios) Co-authored-by: Liudmila Molkova <neskazu@gmail.com> Assisted-by: Claude Sonnet 5

…se and tool-calling conformance scenarios Adds InferenceBetaRawResponseScenario, InferenceBetaRawResponseStreamingScenario, and ToolCallingBetaScenario, closing the beta.messages coverage gap against the existing non-beta conformance scenarios. InferenceBetaStreamingScenario moves out of inference_beta.py into its own inference_streaming_beta.py, matching the inference.py/inference_streaming.py split already used for the non-beta scenarios

…sts.

eternalcuriouslearner · 2026-09-29T21:30:23Z

@benkawecki Sorry for the delay, looking into this.

benkawecki · 2026-10-02T16:18:01Z

@benkawecki Sorry for the delay, looking into this.

Thanks! lmk when you get a chance to take a look.

dskoda1 · 2026-10-02T16:29:46Z

looking forward to this one 🙏

lmolkova

Thanks!

lmolkova · 2026-10-11T19:42:58Z

@benkawecki could you please resolve conflicts and it should be good to go

…ages-create

benkawecki · 2026-10-12T19:53:21Z

@lmolkova conflicts resolved.

lmolkova · 2026-10-13T15:43:09Z

@benkawecki sorry, some conflicts again 😞

lmolkova · 2026-10-13T18:17:52Z

@benkawecki could you please fix tests? there were some changes on main wrt capturing lists and some methods were removed. see #873 for the details

lmolkova · 2026-10-13T19:16:46Z

        args: tuple[Any, ...],
        kwargs: dict[str, Any],
    ) -> MessagesStreamManagerWrapper[Any]:
-        _materialize_tools(kwargs)


why? are these not applicable here? can't input / output messages be iterators / generators here that we consume prematurely?

I missed this when I did my merge. Removed it as it loos like #873 moved to a different pattern. Let me know if my interpretation of that was not correct.

benkawecki · 2026-10-14T15:02:35Z

@lmolkova fixed CI issues.

eternalcuriouslearner · 2026-10-14T15:42:49Z

@benkawecki are we capturing compaction span attribute here?

benkawecki · 2026-10-15T13:05:27Z

@benkawecki are we capturing compaction span attribute here?

I don't think so. I see you have a PR to add compaction attributes. Maybe we should add compaction as a separate follow-on PR to this one?

Copilot AI lite review requested due to automatic review settings September 23, 2026 20:06

benkawecki requested a review from a team as a code owner September 23, 2026 20:06

github-project-automation Bot added this to GenAI Semantic Conventions and Instrumentation libraries Sep 23, 2026

Copilot started reviewing on behalf of benkawecki September 23, 2026 20:06

View session

opentelemetry-pr-dashboard Bot mentioned this pull request Sep 23, 2026

Pull Request Dashboard #171

Open

Copilot AI reviewed Sep 23, 2026

View reviewed changes

benkawecki force-pushed the anthropic-beta-messages-create branch from 1693211 to a9c1ce3 Compare September 23, 2026 20:22

benkawecki and others added 4 commits September 24, 2026 14:03

Address feedback on beta conformance tests not using beta route in te…

bd276a1

…sts.

update PR number of changelog

db81210

benkawecki force-pushed the anthropic-beta-messages-create branch from d7f4340 to db81210 Compare September 24, 2026 18:05

run generate.

db3dee9

This comment has been minimized.

Sign in to view

lmolkova approved these changes Oct 11, 2026

View reviewed changes

Merge remote-tracking branch 'upstream/main' into anthropic-beta-mess…

c74ec95

…ages-create

RichardoMrMu mentioned this pull request Oct 13, 2026

feat(genai-openai): instrument Responses.parse and AsyncResponses.parse #786

Open

Merge branch 'main' into anthropic-beta-messages-create

d3de4ad

fix(anthropic): remove stale stream materialization calls

70da2a0

lmolkova reviewed Oct 13, 2026

View reviewed changes

benkawecki and others added 2 commits October 14, 2026 08:25

fix(anthropic): widen _enter_manager return types for beta streams

6eded70

Merge branch 'main' into anthropic-beta-messages-create

990743e

eternalcuriouslearner mentioned this pull request Oct 14, 2026

[opentelemetry-instrumentation-genai-openai] Record gen_ai.conversation.compacted on Responses spans #900

Open

Conversation

benkawecki commented Sep 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Type of change

How has this been tested?

Checklist

Uh oh!

opentelemetry-pr-dashboard Bot commented Sep 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull request dashboard status

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

benkawecki commented Sep 23, 2026

Uh oh!

aabmass commented Sep 23, 2026

Uh oh!

eternalcuriouslearner commented Sep 29, 2026

Uh oh!

This comment has been minimized.

benkawecki commented Oct 2, 2026

Uh oh!

dskoda1 commented Oct 2, 2026

Uh oh!

lmolkova left a comment

Choose a reason for hiding this comment

Uh oh!

lmolkova commented Oct 11, 2026

Uh oh!

benkawecki commented Oct 12, 2026

Uh oh!

lmolkova commented Oct 13, 2026

Uh oh!

lmolkova commented Oct 13, 2026

Uh oh!

lmolkova Oct 13, 2026

Choose a reason for hiding this comment

Uh oh!

benkawecki Oct 14, 2026

Choose a reason for hiding this comment

Uh oh!

benkawecki commented Oct 14, 2026

Uh oh!

eternalcuriouslearner commented Oct 14, 2026

Uh oh!

benkawecki commented Oct 15, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

benkawecki commented Sep 23, 2026 •

edited

Loading

opentelemetry-pr-dashboard Bot commented Sep 23, 2026 •

edited

Loading