Skip to content

Commit b8f745a

Browse files
xds: Fix TSAN data race on ClientCall cancellation in ExternalProcessorClientInterceptor (grpc#13022)
Prevent concurrent cancellations of the underlying ClientCall in ExternalProcessorClientInterceptor: - Wrap rawCall with SimpleForwardingClientCall using an AtomicBoolean to ensure the underlying ClientCall.cancel() is executed at most once, even if invoked concurrently across threads or from DelayedListener. - In DataPlaneClientCall.cancel(), atomically transition extProcStreamState to FAILED, catch exceptions during onError(), and clear extProcClientCallRequestObserver. - In sendToExtProc(), return early if the ext-proc stream is already completed or the observer is null, and catch unexpected onNext() exceptions to trigger internalOnError() rather than letting exceptions escape into listener callbacks. - Safely complete and clear extProcClientCallRequestObserver in closeExtProcStream() and halfCloseExtProcStream(). - Route all rawCall.cancel() calls in sendMessage(), handleImmediateResponse(), and DataPlaneListener through cancelDownstream(). Fixes the TSAN detected race condition below. The previous fix in commit cf92f2d introduced `AtomicBoolean downstreamCancelled` in `cancelDownstream()`. However, the TSAN data race still reproduced under postsubmit because `DelayedListener` exception path bypasses `cancelDownstream()`: When `DelayedClientCall.start(wrappedListener, headers)` is called, it registers an internal `DelayedListener`. If `wrappedListener.onHeaders()` throws any exception, `DelayedListener.deliverHeaders()` catches it and calls `exceptionThrown()`, which invokes `delayedCall.cancel()` directly. Once `delayedCall` has been activated `(realCall != null)`, `DelayedClientCall.cancel()` directly calls `realCall.cancel()` on whatever thread triggered it without any de-duplication. ``` Details ================== WARNING: ThreadSanitizer: data race (pid=7774) Read of size 1 at 0x000089d8a086 by thread T113: #0 io.grpc.internal.ClientCallImpl.cancelInternal(Ljava/lang/String;Ljava/lang/Throwable;)V ClientCallImpl.java:464 #1 io.grpc.internal.ClientCallImpl.cancel(Ljava/lang/String;Ljava/lang/Throwable;)V ClientCallImpl.java:455 #2 io.grpc.PartialForwardingClientCall.cancel(Ljava/lang/String;Ljava/lang/Throwable;)V PartialForwardingClientCall.java:39 #3 io.grpc.ForwardingClientCall.cancel(Ljava/lang/String;Ljava/lang/Throwable;)V ForwardingClientCall.java:22 grpc#4 io.grpc.ForwardingClientCall$SimpleForwardingClientCall.cancel(Ljava/lang/String;Ljava/lang/Throwable;)V ForwardingClientCall.java:44 grpc#5 io.grpc.PartialForwardingClientCall.cancel(Ljava/lang/String;Ljava/lang/Throwable;)V PartialForwardingClientCall.java:39 grpc#6 io.grpc.ForwardingClientCall.cancel(Ljava/lang/String;Ljava/lang/Throwable;)V ForwardingClientCall.java:22 grpc#7 io.grpc.ForwardingClientCall$SimpleForwardingClientCall.cancel(Ljava/lang/String;Ljava/lang/Throwable;)V ForwardingClientCall.java:44 grpc#8 io.grpc.PartialForwardingClientCall.cancel(Ljava/lang/String;Ljava/lang/Throwable;)V PartialForwardingClientCall.java:39 grpc#9 io.grpc.ForwardingClientCall.cancel(Ljava/lang/String;Ljava/lang/Throwable;)V ForwardingClientCall.java:22 grpc#10 io.grpc.ForwardingClientCall$SimpleForwardingClientCall.cancel(Ljava/lang/String;Ljava/lang/Throwable;)V ForwardingClientCall.java:44 grpc#11 io.grpc.internal.DelayedClientCall$2.run()V DelayedClientCall.java:269 grpc#12 io.grpc.internal.DelayedClientCall.delayOrExecute(Ljava/lang/Runnable;)V DelayedClientCall.java:292 grpc#13 io.grpc.internal.DelayedClientCall.cancel(Lio/grpc/Status;Z)V DelayedClientCall.java:266 grpc#14 io.grpc.internal.DelayedClientCall.cancel(Ljava/lang/String;Ljava/lang/Throwable;)V DelayedClientCall.java:242 grpc#15 io.grpc.xds.ExternalProcessorClientInterceptor$DataPlaneClientCall.internalOnError(Ljava/lang/Throwable;)V ExternalProcessorClientInterceptor.java:715 grpc#16 io.grpc.xds.ExternalProcessorClientInterceptor$DataPlaneClientCall$1.onNext(Lio/envoyproxy/envoy/service/ext_proc/v3/ProcessingResponse;)V ExternalProcessorClientInterceptor.java:457 grpc#17 io.grpc.xds.ExternalProcessorClientInterceptor$DataPlaneClientCall$1.onNext(Ljava/lang/Object;)V ExternalProcessorClientInterceptor.java:426 grpc#18 io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onMessage(Ljava/lang/Object;)V ClientCalls.java:568 grpc#19 io.grpc.ForwardingClientCallListener.onMessage(Ljava/lang/Object;)V ForwardingClientCallListener.java:33 grpc#20 io.grpc.ForwardingClientCallListener.onMessage(Ljava/lang/Object;)V ForwardingClientCallListener.java:33 grpc#21 io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1MessagesAvailable.runInternal()V ClientCallImpl.java:662 grpc#22 io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1MessagesAvailable.runInContext()V ClientCallImpl.java:649 grpc#23 io.grpc.internal.ContextRunnable.run()V ContextRunnable.java:37 grpc#24 io.grpc.internal.SerializingExecutor.run()V SerializingExecutor.java:133 grpc#25 io.grpc.internal.SerializingExecutor.run()V SerializingExecutor.java:133 grpc#26 java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V ThreadPoolExecutor.java:1090 grpc#27 java.util.concurrent.ThreadPoolExecutor$Worker.run()V ThreadPoolExecutor.java:614 grpc#28 java.lang.Thread.runWith(Ljava/lang/Object;Ljava/lang/Runnable;)V Thread.java:1560 grpc#29 java.lang.Thread.run()V Thread.java:1547 grpc#30 (Generated Stub) <null> Previous write of size 1 at 0x000089d8a086 by thread T29: #0 io.grpc.internal.ClientCallImpl.cancelInternal(Ljava/lang/String;Ljava/lang/Throwable;)V ClientCallImpl.java:467 #1 io.grpc.internal.ClientCallImpl.cancel(Ljava/lang/String;Ljava/lang/Throwable;)V ClientCallImpl.java:455 #2 io.grpc.PartialForwardingClientCall.cancel(Ljava/lang/String;Ljava/lang/Throwable;)V PartialForwardingClientCall.java:39 #3 io.grpc.ForwardingClientCall.cancel(Ljava/lang/String;Ljava/lang/Throwable;)V ForwardingClientCall.java:22 grpc#4 io.grpc.ForwardingClientCall$SimpleForwardingClientCall.cancel(Ljava/lang/String;Ljava/lang/Throwable;)V ForwardingClientCall.java:44 grpc#5 io.grpc.PartialForwardingClientCall.cancel(Ljava/lang/String;Ljava/lang/Throwable;)V PartialForwardingClientCall.java:39 grpc#6 io.grpc.ForwardingClientCall.cancel(Ljava/lang/String;Ljava/lang/Throwable;)V ForwardingClientCall.java:22 grpc#7 io.grpc.ForwardingClientCall$SimpleForwardingClientCall.cancel(Ljava/lang/String;Ljava/lang/Throwable;)V ForwardingClientCall.java:44 grpc#8 io.grpc.PartialForwardingClientCall.cancel(Ljava/lang/String;Ljava/lang/Throwable;)V PartialForwardingClientCall.java:39 grpc#9 io.grpc.ForwardingClientCall.cancel(Ljava/lang/String;Ljava/lang/Throwable;)V ForwardingClientCall.java:22 grpc#10 io.grpc.ForwardingClientCall$SimpleForwardingClientCall.cancel(Ljava/lang/String;Ljava/lang/Throwable;)V ForwardingClientCall.java:44 grpc#11 io.grpc.internal.DelayedClientCall$2.run()V DelayedClientCall.java:269 grpc#12 io.grpc.internal.DelayedClientCall.delayOrExecute(Ljava/lang/Runnable;)V DelayedClientCall.java:292 grpc#13 io.grpc.internal.DelayedClientCall.cancel(Lio/grpc/Status;Z)V DelayedClientCall.java:266 grpc#14 io.grpc.internal.DelayedClientCall.cancel(Ljava/lang/String;Ljava/lang/Throwable;)V DelayedClientCall.java:242 grpc#15 io.grpc.PartialForwardingClientCall.cancel(Ljava/lang/String;Ljava/lang/Throwable;)V PartialForwardingClientCall.java:39 grpc#16 io.grpc.ForwardingClientCall.cancel(Ljava/lang/String;Ljava/lang/Throwable;)V ForwardingClientCall.java:22 grpc#17 io.grpc.ForwardingClientCall$SimpleForwardingClientCall.cancel(Ljava/lang/String;Ljava/lang/Throwable;)V ForwardingClientCall.java:44 grpc#18 io.grpc.xds.ExternalProcessorClientInterceptor$DataPlaneClientCall.cancel(Ljava/lang/String;Ljava/lang/Throwable;)V ExternalProcessorClientInterceptor.java:887 grpc#19 io.grpc.xds.XdsNameResolver$RawMessageClientInterceptor$2.cancel(Ljava/lang/String;Ljava/lang/Throwable;)V XdsNameResolver.java:1182 grpc#20 io.grpc.xds.ExternalProcessorClientInterceptorTest.clientInterceptor_contextPropagatedToStartCall()V ExternalProcessorClientInterceptorTest.java:13411 ```
1 parent 64daddc commit b8f745a

2 files changed

Lines changed: 201 additions & 33 deletions

File tree

‎xds/src/main/java/io/grpc/xds/ExternalProcessorClientInterceptor.java‎

Lines changed: 41 additions & 33 deletions
Original file line numberDiff line numberDiff line change
@@ -236,8 +236,18 @@ public <ReqT, RespT> ClientCall<ReqT, RespT> interceptCall(
236236
MethodDescriptor<InputStream, InputStream> rawMethod =
237237
(MethodDescriptor<InputStream, InputStream>) (MethodDescriptor<?, ?>) method;
238238
ClientCall<InputStream, InputStream> rawCall =
239-
(ClientCall<InputStream, InputStream>) (ClientCall<?, ?>)
240-
next.newCall(method, callOptions);
239+
new SimpleForwardingClientCall<InputStream, InputStream>(
240+
(ClientCall<InputStream, InputStream>) (ClientCall<?, ?>)
241+
next.newCall(method, callOptions)) {
242+
private final AtomicBoolean cancelled = new AtomicBoolean(false);
243+
244+
@Override
245+
public void cancel(@Nullable String message, @Nullable Throwable cause) {
246+
if (cancelled.compareAndSet(false, true)) {
247+
super.cancel(message, cause);
248+
}
249+
}
250+
};
241251

242252
// Create a local subclass instance to buffer outbound actions
243253
DataPlaneDelayedCall<InputStream, InputStream> delayedCall =
@@ -362,7 +372,6 @@ private static class DataPlaneClientCall
362372
final AtomicBoolean isProcessingTrailers = new AtomicBoolean(false);
363373
final AtomicBoolean pendingHalfClose = new AtomicBoolean(false);
364374
final AtomicBoolean bodyMessageSentToExtProc = new AtomicBoolean(false);
365-
private final AtomicBoolean downstreamCancelled = new AtomicBoolean(false);
366375

367376
protected DataPlaneClientCall(
368377
DataPlaneDelayedCall<InputStream, InputStream> delayedCall,
@@ -443,13 +452,14 @@ private boolean validateCompressionSupport(BodyResponse bodyResponse) {
443452
.withDescription("gRPC message compression not supported in ext_proc")
444453
.asRuntimeException();
445454
synchronized (streamLock) {
446-
if (!extProcStreamState.get().isCompleted()
447-
&& extProcClientCallRequestObserver != null) {
448-
extProcClientCallRequestObserver.onError(ex);
455+
if (markExtProcStreamFailed(extProcStreamState)) {
456+
if (extProcClientCallRequestObserver != null) {
457+
extProcClientCallRequestObserver.onError(ex);
458+
extProcClientCallRequestObserver = null;
459+
}
449460
}
450461
}
451462
activateCall();
452-
markExtProcStreamFailed(extProcStreamState);
453463
cancelDownstream("gRPC message compression not supported in ext_proc", ex);
454464
closeExtProcStream();
455465
return false;
@@ -463,7 +473,7 @@ public void start(Listener<InputStream> responseListener, Metadata headers) {
463473
this.callContext = Context.current();
464474
clientHeadersStartNanos = System.nanoTime();
465475
this.requestHeaders = headers;
466-
this.wrappedListener = new DataPlaneListener(responseListener, rawCall, this);
476+
this.wrappedListener = new DataPlaneListener(responseListener, this);
467477

468478
// DelayedClientCall.start will buffer the listener and headers until setCall is called.
469479
super.start(wrappedListener, headers);
@@ -664,6 +674,9 @@ public void onError(Throwable t) {
664674
@Override
665675
public void onCompleted() {
666676
if (markExtProcStreamCompleted(extProcStreamState)) {
677+
synchronized (streamLock) {
678+
extProcClientCallRequestObserver = null;
679+
}
667680
handleFailOpen(wrappedListener);
668681
}
669682
}
@@ -693,7 +706,7 @@ public void onCompleted() {
693706

694707
private void sendToExtProc(ProcessingRequest request) {
695708
synchronized (streamLock) {
696-
if (extProcStreamState.get().isCompleted()) {
709+
if (extProcStreamState.get().isCompleted() || extProcClientCallRequestObserver == null) {
697710
return;
698711
}
699712

@@ -830,6 +843,7 @@ private void closeExtProcStream() {
830843
if (markExtProcStreamCompleted(extProcStreamState)) {
831844
if (extProcClientCallRequestObserver != null) {
832845
extProcClientCallRequestObserver.onCompleted();
846+
extProcClientCallRequestObserver = null;
833847
}
834848
}
835849
}
@@ -839,11 +853,7 @@ private void internalOnError(Throwable t) {
839853
if (markExtProcStreamFailed(extProcStreamState)) {
840854
synchronized (streamLock) {
841855
if (extProcClientCallRequestObserver != null) {
842-
try {
843-
extProcClientCallRequestObserver.onError(t);
844-
} catch (Throwable ignored) {
845-
// Ignore exceptions during cancel/onError propagation
846-
}
856+
extProcClientCallRequestObserver.onError(t);
847857
extProcClientCallRequestObserver = null;
848858
}
849859
}
@@ -999,7 +1009,7 @@ public void sendMessage(InputStream message) {
9991009
ByteString copiedBody = ByteString.readFrom(message);
10001010
pendingDrainingMessages.add(new KnownLengthInputStream(copiedBody));
10011011
} catch (IOException e) {
1002-
rawCall.cancel("Failed to copy outbound message for buffering", e);
1012+
cancelDownstream("Failed to copy outbound message for buffering", e);
10031013
}
10041014
return;
10051015
}
@@ -1029,7 +1039,7 @@ public void sendMessage(InputStream message) {
10291039
}
10301040
}
10311041
} catch (IOException e) {
1032-
rawCall.cancel("Failed to serialize message for External Processor", e);
1042+
cancelDownstream("Failed to serialize message for External Processor", e);
10331043
}
10341044
}
10351045
}
@@ -1131,21 +1141,22 @@ public void halfClose() {
11311141
}
11321142
}
11331143

1134-
private void cancelDownstream(@Nullable String message, @Nullable Throwable cause) {
1135-
if (downstreamCancelled.compareAndSet(false, true)) {
1136-
delayedCall.cancel(message, cause);
1137-
}
1144+
void cancelDownstream(@Nullable String message, @Nullable Throwable cause) {
1145+
delayedCall.cancel(message, cause);
11381146
}
11391147

11401148
@Override
11411149
public void cancel(@Nullable String message, @Nullable Throwable cause) {
11421150
synchronized (streamLock) {
1143-
if (!extProcStreamState.get().isCompleted() && extProcClientCallRequestObserver != null) {
1144-
extProcClientCallRequestObserver.onError(
1145-
Status.CANCELLED
1146-
.withDescription(message)
1147-
.withCause(cause)
1148-
.asRuntimeException());
1151+
if (markExtProcStreamFailed(extProcStreamState)) {
1152+
if (extProcClientCallRequestObserver != null) {
1153+
extProcClientCallRequestObserver.onError(
1154+
Status.CANCELLED
1155+
.withDescription(message)
1156+
.withCause(cause)
1157+
.asRuntimeException());
1158+
extProcClientCallRequestObserver = null;
1159+
}
11491160
}
11501161
}
11511162
cancelDownstream(message, cause);
@@ -1318,7 +1329,7 @@ private void handleImmediateResponse(ImmediateResponse immediate, DataPlaneListe
13181329
// If sent in response to any other event, it will cause the data plane RPC to
13191330
// immediately fail with the specified status as if it were an out-of-band
13201331
// cancellation.
1321-
rawCall.cancel(status.getDescription(), null);
1332+
cancelDownstream(status.getDescription(), null);
13221333
listener.unblockAfterStreamComplete();
13231334
}
13241335
closeExtProcStream();
@@ -1442,7 +1453,6 @@ AtomicBoolean getIsProcessingTrailers() {
14421453
}
14431454

14441455
private static class DataPlaneListener extends SimpleForwardingClientCallListener<InputStream> {
1445-
private final ClientCall<?, ?> rawCall;
14461456
private final DataPlaneClientCall dataPlaneClientCall;
14471457
// Path 3: Upstream response bodies queued because upstream to sidestream window not available,
14481458
// response headers not cleared by ext_proc or ext_proc stream draining
@@ -1457,10 +1467,8 @@ private static class DataPlaneListener extends SimpleForwardingClientCallListene
14571467

14581468
protected DataPlaneListener(
14591469
ClientCall.Listener<InputStream> delegate,
1460-
ClientCall<?, ?> rawCall,
14611470
DataPlaneClientCall dataPlaneClientCall) {
14621471
super(delegate);
1463-
this.rawCall = rawCall;
14641472
this.dataPlaneClientCall = dataPlaneClientCall;
14651473
}
14661474

@@ -1538,7 +1546,7 @@ public void onMessage(InputStream message) {
15381546
ByteString copiedBody = ByteString.readFrom(message);
15391547
savedMessages.add(new KnownLengthInputStream(copiedBody));
15401548
} catch (IOException e) {
1541-
rawCall.cancel("Failed to copy inbound message for buffering", e);
1549+
dataPlaneClientCall.cancelDownstream("Failed to copy inbound message for buffering", e);
15421550
}
15431551
return;
15441552
}
@@ -1574,7 +1582,7 @@ public void onMessage(InputStream message) {
15741582
dataPlaneClientCall.drainPendingRequests();
15751583
}
15761584
} catch (IOException e) {
1577-
rawCall.cancel("Failed to read server response", e);
1585+
dataPlaneClientCall.cancelDownstream("Failed to read server response", e);
15781586
}
15791587
}
15801588
}
@@ -1592,7 +1600,7 @@ void drainSavedMessages() {
15921600
sendResponseBodyToExtProc(bodyByteString, false);
15931601
dataPlaneClientCall.bodyMessageSentToExtProc.set(true);
15941602
} catch (IOException e) {
1595-
rawCall.cancel("Failed to read buffered response body", e);
1603+
dataPlaneClientCall.cancelDownstream("Failed to read buffered response body", e);
15961604
}
15971605
}
15981606
}

‎xds/src/test/java/io/grpc/xds/ExternalProcessorClientInterceptorTest.java‎

Lines changed: 160 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -90,9 +90,11 @@
9090
import java.util.List;
9191
import java.util.concurrent.CopyOnWriteArrayList;
9292
import java.util.concurrent.CountDownLatch;
93+
import java.util.concurrent.CyclicBarrier;
9394
import java.util.concurrent.Executor;
9495
import java.util.concurrent.ExecutorService;
9596
import java.util.concurrent.Executors;
97+
import java.util.concurrent.Future;
9698
import java.util.concurrent.ScheduledExecutorService;
9799
import java.util.concurrent.TimeUnit;
98100
import java.util.concurrent.atomic.AtomicBoolean;
@@ -21129,6 +21131,164 @@ public void onClose(Status status, Metadata trailers) {
2112921131
channelManager.close();
2113021132
}
2113121133

21134+
@Test
21135+
public void whenCallCancelledConcurrently_underlyingCallCancelledExactlyOnceAndNoExtProcLeak()
21136+
throws Exception {
21137+
String uniqueExtProcServerName = InProcessServerBuilder.generateName();
21138+
String uniqueDataPlaneServerName = InProcessServerBuilder.generateName();
21139+
21140+
ExternalProcessor proto = createBaseProto(uniqueExtProcServerName)
21141+
.setProcessingMode(ProcessingMode.newBuilder()
21142+
.setRequestHeaderMode(ProcessingMode.HeaderSendMode.SEND)
21143+
.setResponseHeaderMode(ProcessingMode.HeaderSendMode.SEND)
21144+
.build())
21145+
.build();
21146+
ExternalProcessorFilterConfig filterConfig =
21147+
provider.parseFilterConfig(Any.pack(proto), filterContext).config;
21148+
21149+
final AtomicInteger extProcErrorCount = new AtomicInteger();
21150+
final AtomicInteger extProcRequestsAfterError = new AtomicInteger();
21151+
final AtomicBoolean extProcHadError = new AtomicBoolean(false);
21152+
21153+
ExternalProcessorGrpc.ExternalProcessorImplBase extProcImpl =
21154+
new ExternalProcessorGrpc.ExternalProcessorImplBase() {
21155+
@Override
21156+
public StreamObserver<ProcessingRequest> process(
21157+
StreamObserver<ProcessingResponse> responseObserver) {
21158+
return new StreamObserver<ProcessingRequest>() {
21159+
@Override
21160+
public void onNext(ProcessingRequest request) {
21161+
if (extProcHadError.get()) {
21162+
extProcRequestsAfterError.incrementAndGet();
21163+
}
21164+
if (request.hasRequestHeaders()) {
21165+
responseObserver.onNext(ProcessingResponse.newBuilder()
21166+
.setRequestHeaders(HeadersResponse.newBuilder().build())
21167+
.build());
21168+
}
21169+
}
21170+
21171+
@Override
21172+
public void onError(Throwable t) {
21173+
extProcHadError.set(true);
21174+
extProcErrorCount.incrementAndGet();
21175+
}
21176+
21177+
@Override
21178+
public void onCompleted() {
21179+
responseObserver.onCompleted();
21180+
}
21181+
};
21182+
}
21183+
};
21184+
21185+
grpcCleanup.register(InProcessServerBuilder.forName(uniqueExtProcServerName)
21186+
.addService(extProcImpl)
21187+
.directExecutor()
21188+
.build().start());
21189+
21190+
CachedChannelManager channelManager = new CachedChannelManager(config -> {
21191+
return grpcCleanup.register(
21192+
InProcessChannelBuilder.forName(uniqueExtProcServerName).directExecutor().build());
21193+
});
21194+
21195+
ExternalProcessorClientInterceptor interceptor = new ExternalProcessorClientInterceptor(
21196+
filterConfig, channelManager, scheduler, FAKE_CONTEXT);
21197+
21198+
final CountDownLatch dataPlaneServerStartedLatch = new CountDownLatch(1);
21199+
final CountDownLatch releaseServerResponseLatch = new CountDownLatch(1);
21200+
21201+
MutableHandlerRegistry dataPlaneRegistry = new MutableHandlerRegistry();
21202+
dataPlaneRegistry.addService(ServerServiceDefinition.builder("test.TestService")
21203+
.addMethod(METHOD_SAY_HELLO, ServerCalls.asyncUnaryCall((request, responseObserver) -> {
21204+
dataPlaneServerStartedLatch.countDown();
21205+
try {
21206+
releaseServerResponseLatch.await(5, TimeUnit.SECONDS);
21207+
} catch (InterruptedException e) {
21208+
Thread.currentThread().interrupt();
21209+
}
21210+
responseObserver.onNext("Hello " + request);
21211+
responseObserver.onCompleted();
21212+
}))
21213+
.build());
21214+
21215+
grpcCleanup.register(InProcessServerBuilder.forName(uniqueDataPlaneServerName)
21216+
.fallbackHandlerRegistry(dataPlaneRegistry)
21217+
.directExecutor()
21218+
.build().start());
21219+
21220+
final AtomicInteger rawCallCancelCount = new AtomicInteger();
21221+
ClientInterceptor countingInterceptor = new ClientInterceptor() {
21222+
@Override
21223+
public <ReqT, RespT> ClientCall<ReqT, RespT> interceptCall(
21224+
MethodDescriptor<ReqT, RespT> method, CallOptions callOptions, Channel next) {
21225+
return new SimpleForwardingClientCall<ReqT, RespT>(next.newCall(method, callOptions)) {
21226+
@Override
21227+
public void cancel(String message, Throwable cause) {
21228+
rawCallCancelCount.incrementAndGet();
21229+
super.cancel(message, cause);
21230+
}
21231+
};
21232+
}
21233+
};
21234+
21235+
ManagedChannel dataPlaneChannel = grpcCleanup.register(
21236+
InProcessChannelBuilder.forName(uniqueDataPlaneServerName)
21237+
.intercept(countingInterceptor)
21238+
.directExecutor()
21239+
.build());
21240+
21241+
ClientCall<String, String> proxyCall = interceptCall(
21242+
interceptor,
21243+
METHOD_SAY_HELLO,
21244+
DEFAULT_CALL_OPTIONS.withExecutor(MoreExecutors.directExecutor()),
21245+
dataPlaneChannel);
21246+
21247+
proxyCall.start(new ClientCall.Listener<String>() {}, new Metadata());
21248+
21249+
proxyCall.request(1);
21250+
proxyCall.sendMessage("ping");
21251+
proxyCall.halfClose();
21252+
21253+
// Wait until the data plane call has started and is active
21254+
assertThat(dataPlaneServerStartedLatch.await(5, TimeUnit.SECONDS)).isTrue();
21255+
21256+
// Concurrently release server response and trigger multiple concurrent cancellations
21257+
int numCancelThreads = 8;
21258+
ExecutorService executor = Executors.newFixedThreadPool(numCancelThreads + 1);
21259+
CyclicBarrier barrier = new CyclicBarrier(numCancelThreads + 1);
21260+
List<Future<?>> futures = new ArrayList<>();
21261+
21262+
futures.add(executor.submit(() -> {
21263+
barrier.await();
21264+
releaseServerResponseLatch.countDown();
21265+
return null;
21266+
}));
21267+
21268+
for (int i = 0; i < numCancelThreads; i++) {
21269+
final int threadId = i;
21270+
futures.add(executor.submit(() -> {
21271+
barrier.await();
21272+
proxyCall.cancel("Cancel from thread " + threadId, null);
21273+
return null;
21274+
}));
21275+
}
21276+
21277+
for (Future<?> f : futures) {
21278+
f.get(5, TimeUnit.SECONDS);
21279+
}
21280+
21281+
// Underlying rawCall was cancelled exactly once despite multiple concurrent cancellations
21282+
assertThat(rawCallCancelCount.get()).isEqualTo(1);
21283+
// Ext-proc received onError at most once
21284+
assertThat(extProcErrorCount.get()).isAtMost(1);
21285+
// Ext-proc never received any requests after being cancelled
21286+
assertThat(extProcRequestsAfterError.get()).isEqualTo(0);
21287+
21288+
channelManager.close();
21289+
shutdownAndAwaitTermination(executor);
21290+
}
21291+
2113221292
private static List<ProcessingRequest> filterClientRequests(List<ProcessingRequest> requests) {
2113321293
List<ProcessingRequest> clientRequests = new ArrayList<>();
2113421294
for (ProcessingRequest r : requests) {

0 commit comments

Comments
 (0)