Repository navigation
Stop using OpenTelemetry SDK internal AttributesProcessor for pre-aggregated metrics - #4894
Draft
Rajkumar Rangaraj (rajkumar-rangaraj) wants to merge 3 commits into
Draft
Rajkumar Rangaraj (rajkumar-rangaraj) wants to merge 3 commits into
Rajkumar Rangaraj (rajkumar-rangaraj) wants to merge 3 commits into
Conversation
…regated metrics AttributesProcessor and ViewBuilder.addAttributesProcessor() were removed in OpenTelemetry SDK 1.67.0 (open-telemetry/opentelemetry-java#8818), which breaks the build for the OpenTelemetry dependency updates (#4876, #4888). Add the Application Insights specific attributes (connection string and role name overrides, synthetic) and mark the span as pre-aggregated from an InstrumenterCustomizer instead, and limit the metric attributes using the public View attribute filter. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot started reviewing on behalf of
Rajkumar Rangaraj (rajkumar-rangaraj)
November 12, 2026 23:19
- Don't customize rpc instrumenters when only stable rpc semconv is emitted, since rpc.*.call.duration are not pre-aggregated standard metrics - Only mark the span when the metrics state was created by the current operation, not inherited from a parent operation - Don't use the static logger from the enum constructors, it isn't initialized yet - Add tests using real Instrumenters Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
HTTP/RPC spans can be marked pre-aggregated without replacement duration metrics, causing standard-metric loss in nested-operation, RPC opt-in, and no-reader configurations.
4 open issues
What changed in this PR
Refactors Application Insights’ pre-aggregated HTTP/RPC metrics to avoid removed OpenTelemetry SDK internals and help unblock dependency upgrade #4888.
Changes:
- Replaces internal attribute processing with public view filters and instrumenter hooks.
- Moves metric helpers into an agent-owned package and adds operation-state detection.
- Extends tests for synthetic traffic, overrides, missing duration metrics, and reflection resolution.
| File | Description |
|---|---|
agent/agent-tooling/src/test/java/com/microsoft/applicationinsights/agent/internal/preagg/PreAggregatedMetricsTest.java |
Relocates and extends pre-aggregated metric tests. |
agent/agent-tooling/src/main/java/io/opentelemetry/sdk/metrics/ViewBuilderAccessor.java |
Removes the SDK access shim. |
agent/agent-tooling/src/main/java/io/opentelemetry/sdk/metrics/internal/view/MetricViewAttributesProcessor.java |
Removes the SDK-internal attribute processor. |
agent/agent-tooling/src/main/java/io/opentelemetry/sdk/metrics/internal/view/AiViewRegistry.java |
Removes the old SDK-package view registry. |
agent/agent-tooling/src/main/java/com/microsoft/applicationinsights/agent/internal/preagg/UserAgents.java |
Moves synthetic user-agent detection into the agent package. |
agent/agent-tooling/src/main/java/com/microsoft/applicationinsights/agent/internal/preagg/PreAggregatedMetricsCustomizer.java |
Adds metric enrichment and span-marking hooks. |
agent/agent-tooling/src/main/java/com/microsoft/applicationinsights/agent/internal/preagg/MetricView.java |
Defines retained metric and internal attribute keys. |
agent/agent-tooling/src/main/java/com/microsoft/applicationinsights/agent/internal/preagg/MetricsState.java |
Detects upstream operation state through reflection. |
agent/agent-tooling/src/main/java/com/microsoft/applicationinsights/agent/internal/preagg/AiViewRegistry.java |
Registers views using public attribute filters. |
agent/agent-tooling/src/main/java/com/microsoft/applicationinsights/agent/internal/init/SecondEntryPoint.java |
Imports the relocated view registry. |
agent/agent-tooling/src/main/java/com/microsoft/applicationinsights/agent/internal/init/AiInstrumenterCustomizerProvider.java |
Registers metric customization when pre-aggregation is enabled. |
Review effort: Max
| // fall back to assuming the metric will be recorded | ||
| return true; | ||
| } | ||
| return context.get(contextKey) != null; |
| // fall back to assuming the metric will be recorded | ||
| return true; | ||
| } | ||
| return context.get(contextKey) != null; |
|
|
||
| // this is needed for detecting telemetry signals that will trigger pre-aggregated metrics via | ||
| // auto instrumentations | ||
| Span.fromContext(context).setAttribute(AiSemanticAttributes.IS_PRE_AGGREGATED, true); |
| field.setAccessible(true); | ||
| return (ContextKey<?>) field.get(null); | ||
| } catch (ReflectiveOperationException | RuntimeException e) { | ||
| logger.warn( |
…ter is used With otel.metrics.exporter=none (or otlp only), the pre-aggregated metrics don't reach Application Insights, so the spans must not be marked as pre-aggregated, otherwise the standard metrics are not generated from them either. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fix # .
Unblocks #4888 (and the closed #4876): OpenTelemetry SDK 1.67.0 removed the experimental
AttributesProcessor/ViewBuilder.addAttributesProcessor()(open-telemetry/opentelemetry-java#8818). The agent was using them, by putting classes in the SDK'sio.opentelemetry.sdk.metricspackages, to add Application Insights specific attributes to the pre-aggregated standard metrics, so:agent:agent-tooling:compileJavafails and all CI jobs fail on those dependency updates.Changes
The processor had access to the measurement context and did 4 things for the
http.{server,client}.request.durationandrpc.{server,client}.durationmetrics. They are now done as follows:AttributesProcessor)View.builder().setAttributeFilter(...)connectionStringOverrides/roleNameOverrides/ Classic SDK attributesPreAggregatedMetricsCustomizerattributes extractoronEnd()(from context)is_synthetic(http server)user_agent.original)onStart()(sees all start attributes) + attributes extractoronEnd()IS_PRE_AGGREGATEDonEnd(), only when the duration metric will be recorded (see below)PreAggregatedMetricsCustomizeris registered from the existingAiInstrumenterCustomizerProviderfor instrumenters withhasType(HTTP_SERVER | HTTP_CLIENT | RPC_SERVER | RPC_CLIENT), only whenpreAggregatedStandardMetricsis enabled and the Azure Monitor metric exporter is used. Withotel.metrics.exporter=nonethe SDK has no metric readers, so the duration metrics aren't collected; withotlponly, they don't reach Application Insights. In both cases the spans must stay unmarked so ingestion still generates the standard metrics from them. The exporter is created when the SDK is built, which is before any instrumenter is built. RPC instrumenters are only customized when the old RPC semconv is emitted. Withotel.semconv-stability.opt-in=rpc, upstream recordsrpc.*.call.duration, which aren't pre-aggregated standard metrics, so marking those spans would remove them from standard metrics.Instrumenter.doStart()runs context customizers before the operation listeners'onStart(), anddoEnd()runs attributes extractors right before the operation listeners'onEnd()with the same context. So the upstream metrics record the duration with the added attributes.AiViewRegistry,MetricViewandUserAgentstocom.microsoft.applicationinsights.agent.internal.preagg, and deletedViewBuilderAccessorandMetricViewAttributesProcessor. Nothing in the agent lives in the SDK packages anymore.Review focus:
MetricsState(reflection)Some upstream instrumenters have http/rpc attributes but don't record the duration metrics. In v2.32.0: AWS SDK 1.11/2.2 (http + rpc client), RMI, Spring RMI, GWT and the Netty connection spans. Previously those spans were never marked
IS_PRE_AGGREGATEDbecause no measurement was recorded for them. Marking them based onhasType()alone would make the exporter skip them, and ingestion would no longer count them in the standard dependency metrics.To keep exact parity,
MetricsStatechecks whether the upstreamHttp{Server,Client}Metrics/Rpc{Server,Client}Metricsper-operation state is in the context, since that's exactly when they record the duration:metricsStateIsResolvedfails if they're renamed in a future update.Alternatives considered: recording our own duration metrics (larger change, behavior change for AWS SDK/RMI) or wrapping the global
MeterProvider(fragile in the javaagent). Open to other ideas.Known limitations
IS_SYNTHETICis now also set on http server spans. The exporter ignoresapplicationinsights.internal.*span attributes (SpanDataMapper.ignorePrefix), so there's no telemetry change. The internal attributes also reach other metrics on the same instrumenter, such as rpc request/response size, but those have attributes advice, so the SDK drops non-advised keys.Notes for #4888
agent-toolingcompiles and thepreaggtests pass. 35 other unit tests fail withPropagatedSpan cannot be cast to ReadableSpan(SpanExporterWithAttributeProcessorTest,ExporterWithSpanProcessorTest,LogExporterWithAttributeProcessorTest,AlertTriggerSpanProcessorTest). Those tests useOpenTelemetrySdk.builder().build()without a tracer provider, so they get no-op spans on the new SDK and need an explicitSdkTracerProvider. That's not related to this change, so it's left for Bump the opentelemetry group across 1 directory with 11 updates #4888.rpc.{client,server}.call.durationfor the new RPC semconv, whichMetricViewand the exporter don't cover yet. Until they do, rpc spans aren't marked in stable-only mode (see above).Validation (on current main versions: SDK 1.66.0, instrumentation 2.32.0)
./gradlew :agent:agent-tooling:check: passed (334 tests, 0 failures)./gradlew :agent:agent-tooling:test --tests "*preagg*": passed (20 tests)PreAggregatedMetricsInstrumenterTest(6) uses realInstrumenters with the customizer applied through the customizer provider mechanism: http server (synthetic + override attributes, span marked), rpc client with metrics, rpc client without metrics (RMI/AWS SDK case, not marked), stable-only rpc semconv (not customized), and with/without the Azure Monitor metric exporter.PreAggregatedMetricsTest(14) covers the existing http client/server and rpc cases, a span not marked when no metric is recorded (each type), a nested span inheriting its parent's metrics state (not marked), reflection resolution, and a missing field falling back without failing../gradlew :smoke-tests:apps:HttpPreaggregatedMetrics:smokeTest --tests "*HttpPreaggregatedMetricsTest$Tomcat8Java8Test": passed./gradlew :smoke-tests:apps:HttpPreaggregatedMetrics:smokeTest --tests "*HttpPreaggregatedMetricsTestOtlpEnabled$Tomcat8Java8Test": passed./gradlew :smoke-tests:apps:PreAggMetricsWithRoleNameOverridesAndSampling:smokeTest --tests "*PreAggMetricsWithRoleNameOverridesAndSamplingTest$Tomcat8Java8Test": passed./gradlew :smoke-tests:apps:gRPC:smokeTest --tests "*GrpcTest$Java8Test": passed./gradlew :smoke-tests:apps:ConnectionStringOverrides:smokeTest --tests "*ConnectionStringOverridesTest$Tomcat8Java8Test": passed./gradlew :agent:agent-tooling:test --tests "*preagg*"passes (18/18, run before the Azure Monitor metric exporter gate was added). Other failures are listed above.For significant contributions please make sure you have completed the following items: