Skip to content

Commit 60bc55f

Browse files
committed
feat!(opentelemetry): add push notification tracing and trace-context propagation
- PushNotificationSender.sendNotification() returns boolean to let decorators observe delivery outcome - OpenTelemetryPushNotificationSenderDecorator wraps sendNotification() with a span and records delivery failures - OpenTelemetryA2AHttpClient injects W3C trace-context headers into outbound push-notification POSTs - @Instrumented CDI qualifier selects a custom A2AHttpClient bean to wrap rather than replacing it Signed-off-by: Emmanuel Hugonnet <ehugonne@redhat.com>
1 parent 05f5d9b commit 60bc55f

39 files changed

Lines changed: 1674 additions & 107 deletions

File tree

‎compat-0.3/server-conversion/src/test/java/org/a2aproject/sdk/compat03/conversion/AbstractA2ARequestHandlerTest_v0_3.java‎

Lines changed: 0 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -30,7 +30,6 @@
3030
import org.a2aproject.sdk.server.tasks.InMemoryPushNotificationConfigStore;
3131
import org.a2aproject.sdk.server.tasks.InMemoryTaskStore;
3232
import org.a2aproject.sdk.server.tasks.PushNotificationConfigStore;
33-
import org.a2aproject.sdk.server.tasks.PushNotificationSender;
3433
import org.a2aproject.sdk.server.tasks.PushNotificationUrlValidator;
3534
import org.a2aproject.sdk.server.tasks.TaskStore;
3635
import org.a2aproject.sdk.spec.A2AError;
@@ -85,8 +84,6 @@ public abstract class AbstractA2ARequestHandlerTest_v0_3 {
8584
.parts(new TextPart_v0_3("test message"))
8685
.build();
8786

88-
private static final PushNotificationSender NOOP_PUSHNOTIFICATION_SENDER = (event, snapshot) -> {};
89-
9087
// V1.0 backend infrastructure
9188
protected AgentExecutor agentExecutor;
9289
protected TaskStore taskStore;

‎docs/content/dev/extra/http-clients.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -266,3 +266,9 @@ public class MyHttpClientProducer {
266266
| 0 | JDK | Always (fallback) |
267267

268268
If the CDI container is active but no `A2AHttpClient` bean is registered, this provider is skipped and the next available provider is used.
269+
270+
### Compatibility with OpenTelemetry
271+
272+
When the [OpenTelemetry server module](opentelemetry.md) is on the classpath alongside a custom `A2AHttpClient` CDI bean, the OpenTelemetry module wraps your bean with trace-context propagation rather than replacing it. Your custom configuration (mTLS, proxy, authentication, timeouts, etc.) is preserved for push notification delivery while `traceparent` and `tracestate` headers are injected automatically.
273+
274+
This works through the `@Instrumented` CDI qualifier (defined in `server-common`): the OpenTelemetry module produces an `@Instrumented A2AHttpClient` that delegates to your unqualified bean. `BasePushNotificationSender` prefers the `@Instrumented` variant when available, so both your custom configuration and the trace-context headers are present on outbound requests. No changes to your producer code are required.

‎docs/content/dev/extra/opentelemetry.md‎

Lines changed: 65 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -106,6 +106,49 @@ The following attributes are automatically added to spans (defined in `A2AObserv
106106
| `gen_ai.agent.a2a.response` | Full response data (only if extraction enabled) |
107107
| `error.type` | Error message (on failures) |
108108

109+
### Push Notification Delivery Tracing
110+
111+
When the server delivers a push notification to a client webhook, `OpenTelemetryPushNotificationSenderDecorator` creates a `CLIENT` span named `SendPushNotification`. This span is independent of the original request span but carries the `gen_ai.agent.a2a.task_id` attribute, allowing correlation by task ID in your observability tool. The span status reflects the actual delivery outcome: `OK` when all notifications succeed, `ERROR` when one or more deliveries fail (with `error.type` set to `delivery_failed`) or when an exception is thrown.
112+
113+
The module also provides **W3C trace-context propagation** for outbound push notification HTTP requests via the `@Instrumented` CDI qualifier and `OpenTelemetryHttpClientProducer`.
114+
115+
#### How it works
116+
117+
The `OpenTelemetryHttpClientProducer` CDI bean produces an `@Instrumented A2AHttpClient` — a wrapper that injects `traceparent` and `tracestate` headers into every outbound HTTP request. `BasePushNotificationSender` resolves the HTTP client in this order:
118+
119+
1. An `@Instrumented A2AHttpClient` bean (if available — e.g. the one produced by this module)
120+
2. An unqualified `A2AHttpClient` bean (if the application provides one)
121+
3. `A2AHttpClientFactory.create()` as last resort
122+
123+
The `@Instrumented` qualifier (defined in `server-common`) marks HTTP client beans that have been wrapped with observability instrumentation. It ensures the instrumented client does not compete with the application's own `A2AHttpClient` bean in default CDI resolution — the two coexist, and the instrumented variant wraps the application's bean as its delegate.
124+
125+
#### Custom HTTP client compatibility
126+
127+
If your application provides a custom `A2AHttpClient` CDI bean (e.g. for mTLS, proxy, or custom timeouts), the OpenTelemetry wrapper **preserves that configuration** by delegating to your bean rather than replacing it. No additional annotations or configuration are needed — just produce your bean as usual:
128+
129+
```java
130+
@Produces
131+
@ApplicationScoped
132+
public A2AHttpClient myHttpClient() {
133+
// your custom client (mTLS, proxy, auth, etc.)
134+
}
135+
```
136+
137+
The OpenTelemetry module automatically wraps it with trace-context injection.
138+
139+
#### Context-propagating executor
140+
141+
The producer also creates a context-propagating `Executor` (via `Context.taskWrapping()`) that ensures the OTel trace context survives the async boundary when push notifications are dispatched to multiple recipients in parallel. This allows the receiving webhook to join the distributed trace.
142+
143+
| Attribute | Description |
144+
|-----------|-------------|
145+
| `gen_ai.agent.a2a.operation.name` | `SendPushNotification` |
146+
| `gen_ai.agent.a2a.task_id` | Task identifier |
147+
| `gen_ai.agent.a2a.context_id` | Context identifier (when available) |
148+
| `gen_ai.agent.a2a.push_notification.event_kind` | The type of streaming event delivered (e.g. `TaskStatusUpdateEvent`) |
149+
| `gen_ai.agent.a2a.response` | Full event payload (only if extraction enabled) |
150+
| `error.type` | `delivery_failed` when one or more deliveries fail, or the exception message when an exception is thrown |
151+
109152
### Request/Response Extraction
110153

111154
Enable request and response data extraction in spans using JVM system properties:
@@ -124,7 +167,7 @@ Enable request and response data extraction in spans using JVM system properties
124167

125168
### Instrumentation
126169

127-
Adds OpenTelemetry spans to A2A client operations:
170+
Adds OpenTelemetry spans and metrics to A2A client operations:
128171

129172
```xml
130173
<dependency>
@@ -133,6 +176,27 @@ Adds OpenTelemetry spans to A2A client operations:
133176
</dependency>
134177
```
135178

179+
To also enable `gen_ai.client.operation.duration` metrics, pass a `Meter` via the transport config:
180+
181+
```java
182+
config.setParameters(Map.of(
183+
OpenTelemetryClientTransportWrapper.OTEL_TRACER_KEY, openTelemetry.getTracer("my-service"),
184+
OpenTelemetryClientTransportWrapper.OTEL_METER_KEY, openTelemetry.getMeter("my-service")
185+
));
186+
```
187+
188+
| Instrument | Type | Unit | Description |
189+
|------------|------|------|-------------|
190+
| `gen_ai.client.operation.duration` | Histogram | `s` | Duration of each A2A client operation call |
191+
192+
Attributes on `gen_ai.client.operation.duration`:
193+
194+
| Attribute | Value |
195+
|-----------|-------|
196+
| `gen_ai.agent.a2a.operation.name` | A2A method name (e.g. `message/send`) |
197+
| `gen_ai.system` | `a2a` |
198+
| `error.type` | `error` (only on failure) |
199+
136200
### Context Propagation
137201

138202
For trace context propagation across async client boundaries:

‎extras/http-client-cdi/src/test/java/org/a2aproject/sdk/client/http/cdi/CdiA2AHttpClientFactoryTest.java‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -81,6 +81,17 @@ public void testCreateDoesNotLogWarningWhenCdiSucceeds() {
8181
assertFalse(records.stream().anyMatch(r -> r.getLevel() == Level.WARNING),
8282
"No warning should be logged when CDI provider succeeds");
8383
}
84+
85+
@Test
86+
public void testCreateExcludingCdiSkipsTheCdiBeanAndFallsBack() {
87+
// Regression guard for the recursive CDI self-delegation bug: a CDI producer that
88+
// produces an A2AHttpClient must call createExcluding("cdi") so the factory does not
89+
// re-enter the container and return the proxy being constructed as its own delegate.
90+
A2AHttpClient client = A2AHttpClientFactory.createExcluding("cdi");
91+
assertNotNull(client);
92+
assertFalse(client instanceof DummyA2AHttpClient,
93+
"createExcluding('cdi') must not return the CDI-registered bean");
94+
}
8495
}
8596

8697
@Nested

‎extras/opentelemetry/common/src/main/java/org/a2aproject/sdk/extras/opentelemetry/A2AObservabilityNames.java‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,4 +20,8 @@ public interface A2AObservabilityNames {
2020
String GENAI_RESPONSE = GENAI_PREFIX + ".response"; // gen_ai.output.messages ?
2121
String GENAI_ROLE = GENAI_PREFIX + ".role";
2222
String GENAI_TASK_ID = GENAI_PREFIX + ".task_id";
23+
24+
// Push notification
25+
String PUSH_NOTIFICATION_EVENT_KIND = GENAI_PREFIX + ".push_notification.event_kind";
26+
String PUSH_NOTIFICATION_DELIVERY_FAILED = "delivery_failed";
2327
}

‎extras/opentelemetry/integration-tests/src/main/java/org/a2aproject/sdk/extras/opentelemetry/it/A2ATestRoutes.java‎

Lines changed: 72 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,7 @@
11
package org.a2aproject.sdk.extras.opentelemetry.it;
22

3-
4-
53
import static io.vertx.core.http.HttpHeaders.CONTENT_TYPE;
4+
import static java.net.HttpURLConnection.HTTP_OK;
65

76
import com.google.gson.Gson;
87
import com.google.gson.GsonBuilder;
@@ -12,6 +11,7 @@
1211
import org.a2aproject.sdk.jsonrpc.common.json.JsonUtil;
1312
import org.a2aproject.sdk.spec.Task;
1413
import org.a2aproject.sdk.spec.TaskArtifactUpdateEvent;
14+
import org.a2aproject.sdk.spec.TaskPushNotificationConfig;
1515
import org.a2aproject.sdk.spec.TaskStatusUpdateEvent;
1616
import io.opentelemetry.api.trace.Span;
1717
import io.opentelemetry.api.trace.Tracer;
@@ -26,7 +26,10 @@
2626
import jakarta.enterprise.inject.Produces;
2727
import jakarta.inject.Inject;
2828
import jakarta.inject.Singleton;
29+
import java.util.HashMap;
2930
import java.util.List;
31+
import java.util.Map;
32+
import java.util.concurrent.CopyOnWriteArrayList;
3033
import java.util.stream.Collectors;
3134

3235
/**
@@ -40,6 +43,11 @@ public class A2ATestRoutes {
4043
private static final String TEXT_PLAIN = "text/plain";
4144
private static final Gson gson = new GsonBuilder().create();
4245

46+
/**
47+
* Captured webhook requests, each entry is a map of header name to value.
48+
*/
49+
private final List<Map<String, String>> webhookRequests = new CopyOnWriteArrayList<>();
50+
4351
@Inject
4452
TestUtilsBean testUtilsBean;
4553
@Inject
@@ -76,6 +84,26 @@ void setupRoutes(@Observes Router router) {
7684
.produces(TEXT_PLAIN)
7785
.handler(ctx -> getChildQueueCount(ctx.pathParam("taskId"), ctx));
7886

87+
router.post("/test/pushconfig/:taskId")
88+
.consumes(APPLICATION_JSON)
89+
.handler(BodyHandler.create())
90+
.blockingHandler(ctx -> savePushNotificationConfig(ctx.pathParam("taskId"), ctx.body().asString(), ctx));
91+
92+
router.delete("/test/pushconfig/:taskId/:configId")
93+
.blockingHandler(ctx -> deletePushNotificationConfig(
94+
ctx.pathParam("taskId"), ctx.pathParam("configId"), ctx));
95+
96+
router.post("/test/webhook")
97+
.handler(BodyHandler.create())
98+
.handler(ctx -> captureWebhook(ctx));
99+
100+
router.get("/test/webhook/requests")
101+
.produces(APPLICATION_JSON)
102+
.handler(ctx -> getWebhookRequests(ctx));
103+
104+
router.delete("/test/webhook/requests")
105+
.handler(ctx -> clearWebhookRequests(ctx));
106+
79107
router.get("/hello")
80108
.produces(TEXT_PLAIN)
81109
.handler(ctx -> hello(ctx));
@@ -180,6 +208,48 @@ public void getChildQueueCount(String taskId, RoutingContext rc) {
180208
.end(String.valueOf(count));
181209
}
182210

211+
public void savePushNotificationConfig(String taskId, String body, RoutingContext rc) {
212+
try {
213+
TaskPushNotificationConfig config = JsonUtil.fromJson(body, TaskPushNotificationConfig.class);
214+
testUtilsBean.savePushNotificationConfig(taskId, config);
215+
rc.response()
216+
.setStatusCode(200)
217+
.end();
218+
} catch (Throwable t) {
219+
errorResponse(t, rc);
220+
}
221+
}
222+
223+
public void deletePushNotificationConfig(String taskId, String configId, RoutingContext rc) {
224+
try {
225+
testUtilsBean.deletePushNotificationConfig(taskId, configId);
226+
rc.response()
227+
.setStatusCode(200)
228+
.end();
229+
} catch (Throwable t) {
230+
errorResponse(t, rc);
231+
}
232+
}
233+
234+
private void captureWebhook(RoutingContext rc) {
235+
Map<String, String> headers = new HashMap<>();
236+
rc.request().headers().forEach(entry -> headers.put(entry.getKey(), entry.getValue()));
237+
webhookRequests.add(headers);
238+
rc.response().setStatusCode(HTTP_OK).end();
239+
}
240+
241+
private void getWebhookRequests(RoutingContext rc) {
242+
rc.response()
243+
.setStatusCode(HTTP_OK)
244+
.putHeader(CONTENT_TYPE, APPLICATION_JSON)
245+
.end(gson.toJson(webhookRequests));
246+
}
247+
248+
private void clearWebhookRequests(RoutingContext rc) {
249+
webhookRequests.clear();
250+
rc.response().setStatusCode(HTTP_OK).end();
251+
}
252+
183253
public void hello(RoutingContext rc) {
184254
Span span = tracer.spanBuilder("hello").startSpan();
185255
try (Scope scope = span.makeCurrent()) {
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
package org.a2aproject.sdk.extras.opentelemetry.it;
2+
3+
import jakarta.enterprise.context.ApplicationScoped;
4+
import jakarta.enterprise.inject.Produces;
5+
import org.a2aproject.sdk.client.http.A2AHttpClient;
6+
import org.a2aproject.sdk.client.http.A2AHttpClientFactory;
7+
8+
/**
9+
* Produces a custom {@link A2AHttpClient} CDI bean that adds a marker header to
10+
* every outbound POST request. Used by integration tests to verify that the
11+
* OpenTelemetry module wraps the application's client instead of replacing it.
12+
*/
13+
@ApplicationScoped
14+
public class CustomHttpClientProducer {
15+
16+
static final String CUSTOM_HEADER_NAME = "X-Custom-A2A-Client";
17+
static final String CUSTOM_HEADER_VALUE = "integration-test";
18+
19+
@Produces
20+
@ApplicationScoped
21+
A2AHttpClient customClient() {
22+
return new MarkerHeaderA2AHttpClient(A2AHttpClientFactory.createExcluding("cdi"));
23+
}
24+
25+
/**
26+
* Delegating {@link A2AHttpClient} that injects a marker header into every
27+
* POST request for test observability.
28+
*/
29+
static class MarkerHeaderA2AHttpClient implements A2AHttpClient {
30+
31+
private final A2AHttpClient delegate;
32+
33+
MarkerHeaderA2AHttpClient(A2AHttpClient delegate) {
34+
this.delegate = delegate;
35+
}
36+
37+
@Override
38+
public GetBuilder createGet() {
39+
return delegate.createGet();
40+
}
41+
42+
@Override
43+
public PostBuilder createPost() {
44+
PostBuilder builder = delegate.createPost();
45+
builder.addHeader(CUSTOM_HEADER_NAME, CUSTOM_HEADER_VALUE);
46+
return builder;
47+
}
48+
49+
@Override
50+
public DeleteBuilder createDelete() {
51+
return delegate.createDelete();
52+
}
53+
}
54+
}

‎extras/opentelemetry/integration-tests/src/main/java/org/a2aproject/sdk/extras/opentelemetry/it/TestAgentCardProducer.java‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -31,7 +31,7 @@ public AgentCard agentCard() {
3131
.documentationUrl("http://example.com/test")
3232
.capabilities(AgentCapabilities.builder()
3333
.streaming(true)
34-
.pushNotifications(false)
34+
.pushNotifications(true)
3535
.build())
3636
.defaultInputModes(Collections.singletonList("text"))
3737
.defaultOutputModes(Collections.singletonList("text"))

‎extras/opentelemetry/integration-tests/src/main/java/org/a2aproject/sdk/extras/opentelemetry/it/TestUtilsBean.java‎

Lines changed: 16 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,15 +1,17 @@
11
package org.a2aproject.sdk.extras.opentelemetry.it;
22

33
import org.a2aproject.sdk.server.events.QueueManager;
4+
import org.a2aproject.sdk.server.tasks.PushNotificationConfigStore;
45
import org.a2aproject.sdk.server.tasks.TaskStore;
56
import org.a2aproject.sdk.spec.Event;
67
import org.a2aproject.sdk.spec.Task;
8+
import org.a2aproject.sdk.spec.TaskPushNotificationConfig;
79
import jakarta.enterprise.context.ApplicationScoped;
810
import jakarta.inject.Inject;
911

1012
/**
1113
* Test utilities for OpenTelemetry integration tests.
12-
* Allows direct manipulation of tasks and queues for testing.
14+
* Allows direct manipulation of tasks, queues, and push notification configs for testing.
1315
*/
1416
@ApplicationScoped
1517
public class TestUtilsBean {
@@ -20,6 +22,9 @@ public class TestUtilsBean {
2022
@Inject
2123
QueueManager queueManager;
2224

25+
@Inject
26+
PushNotificationConfigStore pushNotificationConfigStore;
27+
2328
public void saveTask(Task task) {
2429
taskStore.save(task, false);
2530
}
@@ -43,4 +48,14 @@ public void enqueueEvent(String taskId, Event event) {
4348
public int getChildQueueCount(String taskId) {
4449
return queueManager.getActiveChildQueueCount(taskId);
4550
}
51+
52+
public void savePushNotificationConfig(String taskId, TaskPushNotificationConfig config) {
53+
pushNotificationConfigStore.setInfo(
54+
TaskPushNotificationConfig.builder(config).taskId(taskId).build(),
55+
null);
56+
}
57+
58+
public void deletePushNotificationConfig(String taskId, String configId) {
59+
pushNotificationConfigStore.deleteInfo(taskId, configId);
60+
}
4661
}

‎extras/opentelemetry/integration-tests/src/main/resources/application.properties‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -29,3 +29,7 @@ quarkus.log.category."io.opentelemetry".level=DEBUG
2929

3030
# Authorization is not configured in this test application
3131
a2a.authorization.required=false
32+
33+
# Allow HTTP and localhost for push notification testing
34+
a2a.push-notification.url.allowed-schemes=http,https
35+
a2a.push-notification.url.allow-private-network-targets=true

0 commit comments

Comments
 (0)