Skip to content

Commit 409a33f

Browse files
authored
feat(api): bound field occurrences of all gRPC API requests (#7052)
* feat(api): bound transaction field occurrences before parsing Scan the wire bytes before parsing on the API entry points that accept a binary Transaction, and reject the request once it exceeds 50,000 field occurrences: - TransactionAdmissionGuard counts every tag with CodedInputStream, recurses into known sub-messages by descriptor, and recurses into the contract payload with the type the node unpacks it into, following protobuf merge semantics. Group encoding and nesting deeper than 32 are rejected. Block, database and P2P parsing do not use it. - For gRPC, every method whose request is a Transaction gets a request marshaller that reads the request with the inbound size bound, runs the guard and hands the same bytes to the original marshaller. A rejection returns a sentinel instead of throwing, and a handler wrapper closes the call with INVALID_ARGUMENT before the service method runs. This affects BroadcastTransaction, GetTransactionSignWeight, GetTransactionApprovedList, GetShieldTransactionHash and CreateCommonTransaction. - /wallet/broadcasthex runs the guard before Transaction.parseFrom. - Admission rejections use one fixed client message and one DEBUG line per rejected request, without a stack trace. Across the full mainnet history (15.79 billion transactions), the maximum observed count was 3,926 field occurrences. This is an API admission policy, not a consensus-validity limit: previously tolerated encodings and unusually complex transactions may now be rejected at these entry points. Consensus, execution and block validation are unchanged. * perf(api): preserve KnownLength in transaction admission Use the exact size of KnownLength request streams to allocate the bounded read buffer once, including zero-length requests. Pass checked bytes to the original marshaller through a KnownLength stream to restore its array-backed protobuf parsing fast path. Keep admission budgets, size limits, rejection handling and the original delegate unchanged. Compressed requests retain the bounded growing read path and also use the KnownLength handoff after checking. * feat(api): bound field occurrences across gRPC API requests Apply the 50,000-field-occurrence admission budget to requests of the Wallet, WalletSolidity, Database and Monitor services before the original request parser runs. Previously only Transaction requests had an occurrence budget. Resolve the original marshaller's protobuf type once when wrapping the method and use it as the scan root. Fall back to the proto schema when no protobuf instance is available. Normal rejections use per-method, per-reason sentinels; the schema-only fallback counts and throws the rejection before delegate parsing. Preserve the original marshaller and KnownLength handoff. Transaction requests keep their rejection message; other requests use "request rejected by admission check". Count admission rejections by method and reason through the existing metrics system. Newly guarded methods also reject group encoding and nesting deeper than 32, and report decoding errors as INVALID_ARGUMENT instead of UNKNOWN. Reflection and P2P are unchanged. * fix(api): reject oversized compressed requests without an error log Map the RESOURCE_EXHAUSTED that the grpc decompressing stream throws while the admission marshaller reads a request over the inbound size limit to the TOO_LARGE rejection, and close the call with RESOURCE_EXHAUSTED, the status grpc already gives an uncompressed message over the limit, instead of UNKNOWN. Other rejections keep INVALID_ARGUMENT. * refactor(api): fail at wiring time without a request prototype Every guarded gRPC method must now have a request marshaller that gives a protobuf prototype. GuardedRequestMarshaller.guard takes the scan root from that prototype and throws IllegalStateException with the full method name otherwise, so such a method fails the registration instead of being kept unguarded or guarded through a separate path. Remove the schema-only Plain marshaller, the empty-bytes probe for non-prototype marshallers and the schema lookup in RpcService, and replace their tests with wiring failure tests. * refactor(api): register gRPC services through RpcService Add an abstract services() to RpcService. addServices guards every definition it returns and registers it, so the subclasses only list their services and no longer register them directly. Remove the one-argument guardRequestMethods. Keep the registration order, the per-port service sets and the cursor interceptor that wraps the SOLIDITY and PBFT services before the admission wrapper. The reflection service is still registered in initServerBuilder and is not wrapped. Make the registration and cursor wiring tests call addServices, and make the shared registration check also fail a method that is guarded twice. * refactor(api): tighten gRPC service wiring and admission helpers Make initServerBuilder, addInterceptor and initServer of RpcService protected final, so subclasses declare their services through services() instead of adding them in these steps. GuardedRequestMarshaller.guard now throws IllegalStateException when the request marshaller is already guarded. ServerInterceptors.intercept preserves the request marshaller, so this also detects a guarded definition intercepted before another guard. Turn parseChecked into an instance method, fold rejectionMessage, make prototypeOf private and move the rejection counter into AdmissionCheckedHandler. Remove the test of a plain delegate, which guard no longer accepts, and rename leftover test variables. * refactor(api): rename the admission guard to ProtobufAdmissionGuard The guard scans the requests of every guarded gRPC method and of /wallet/broadcasthex, not only transactions. Rename TransactionAdmissionGuard to ProtobufAdmissionGuard, TransactionAdmissionException to ProtobufAdmissionException and MAX_TX_OCCURRENCES to MAX_OCCURRENCES, with the test class. checkTransaction and countTransaction keep their names. Behavior, messages, status codes and metric labels are unchanged. * refactor(api): clarify admission sentinel naming and buffer growth Rename the rejection map of GuardedRequestMarshaller to rejectionSentinels and its accessor to rejectionSentinel, to distinguish the per-reason sentinel objects from rejection state. Split the buffer growth in readBounded into a doubling step and a cap, and compute the cap once before the loop. The growth is unchanged. Add tests for a negative max and for reading at most one byte past max.
1 parent d7df62b commit 409a33f

21 files changed

Lines changed: 3562 additions & 36 deletions

‎common/src/main/java/org/tron/common/prometheus/MetricKeys.java‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,7 @@ public static class Counter {
2626
public static final String BLOCK_FETCH_ARMED = "tron:block_fetch_armed";
2727
public static final String BLOCK_FETCH_SECONDARY = "tron:block_fetch_secondary";
2828
public static final String BLOCK_ALREADY_KNOWN = "tron:block_already_known";
29+
public static final String GRPC_ADMISSION_REJECT = "tron:grpc_admission_reject";
2930

3031
private Counter() {
3132
throw new IllegalStateException("Counter");

‎common/src/main/java/org/tron/common/prometheus/MetricsCounter.java‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,8 @@ class MetricsCounter {
3030
+ "was already known when the response was handled (best-effort: concurrent "
3131
+ "arrivals may be missed; a duplicate is not attributed to secondary "
3232
+ "fetches).");
33+
init(MetricKeys.Counter.GRPC_ADMISSION_REJECT, "gRPC requests rejected by admission check.",
34+
"method", "reason");
3335
}
3436

3537
private MetricsCounter() {

‎framework/src/main/java/org/tron/common/application/RpcService.java‎

Lines changed: 43 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -16,15 +16,22 @@
1616
package org.tron.common.application;
1717

1818
import io.grpc.Server;
19+
import io.grpc.ServerMethodDefinition;
20+
import io.grpc.ServerServiceDefinition;
21+
import io.grpc.ServiceDescriptor;
1922
import io.grpc.netty.NettyServerBuilder;
2023
import io.grpc.protobuf.services.ProtoReflectionService;
24+
import java.util.ArrayList;
25+
import java.util.List;
2126
import java.util.concurrent.CompletableFuture;
2227
import java.util.concurrent.ExecutorService;
2328
import java.util.concurrent.TimeUnit;
2429
import lombok.extern.slf4j.Slf4j;
2530
import org.springframework.beans.factory.annotation.Autowired;
2631
import org.tron.common.es.ExecutorServiceManager;
2732
import org.tron.common.parameter.CommonParameter;
33+
import org.tron.core.admission.GuardedRequestMarshaller;
34+
import org.tron.core.admission.ProtobufAdmissionGuard;
2835
import org.tron.core.config.args.Args;
2936
import org.tron.core.services.filter.LiteFnQueryGrpcInterceptor;
3037
import org.tron.core.services.ratelimiter.PrometheusInterceptor;
@@ -84,14 +91,14 @@ public void innerStop() throws Exception {
8491
@Override
8592
public CompletableFuture<Boolean> start() {
8693
NettyServerBuilder serverBuilder = initServerBuilder();
87-
addService(serverBuilder);
94+
addServices(serverBuilder);
8895
addInterceptor(serverBuilder);
8996
initServer(serverBuilder);
9097
this.rateLimiterInterceptor.init(this.apiServer);
9198
return super.start();
9299
}
93100

94-
protected NettyServerBuilder initServerBuilder() {
101+
protected final NettyServerBuilder initServerBuilder() {
95102
CommonParameter parameter = Args.getInstance();
96103
NettyServerBuilder serverBuilder = NettyServerBuilder.forPort(this.port);
97104
if (parameter.getRpcThreadNum() > 0) {
@@ -116,9 +123,40 @@ protected NettyServerBuilder initServerBuilder() {
116123
return serverBuilder;
117124
}
118125

119-
protected abstract void addService(NettyServerBuilder serverBuilder);
126+
/** The services this port serves; {@link #addServices} guards and registers each of them. */
127+
protected abstract List<ServerServiceDefinition> services();
120128

121-
protected void addInterceptor(NettyServerBuilder serverBuilder) {
129+
private void addServices(NettyServerBuilder serverBuilder) {
130+
int maxMessageSize = Args.getInstance().getMaxMessageSize();
131+
for (ServerServiceDefinition service : services()) {
132+
serverBuilder.addService(guardRequestMethods(service, maxMessageSize,
133+
ProtobufAdmissionGuard.MAX_OCCURRENCES));
134+
}
135+
}
136+
137+
/** Guards every method; fails if a request marshaller gives no protobuf prototype. */
138+
static ServerServiceDefinition guardRequestMethods(ServerServiceDefinition definition,
139+
int maxMessageSize, int maxOccurrences) {
140+
ServiceDescriptor service = definition.getServiceDescriptor();
141+
ServiceDescriptor.Builder serviceBuilder = ServiceDescriptor.newBuilder(service.getName())
142+
.setSchemaDescriptor(service.getSchemaDescriptor());
143+
List<ServerMethodDefinition<?, ?>> methods = new ArrayList<>();
144+
for (ServerMethodDefinition<?, ?> method : definition.getMethods()) {
145+
methods.add(GuardedRequestMarshaller.guard(method, maxMessageSize, maxOccurrences));
146+
}
147+
// grpc requires the same MethodDescriptor instances in both descriptors.
148+
for (ServerMethodDefinition<?, ?> method : methods) {
149+
serviceBuilder.addMethod(method.getMethodDescriptor());
150+
}
151+
ServerServiceDefinition.Builder builder =
152+
ServerServiceDefinition.builder(serviceBuilder.build());
153+
for (ServerMethodDefinition<?, ?> method : methods) {
154+
builder.addMethod(method);
155+
}
156+
return builder.build();
157+
}
158+
159+
protected final void addInterceptor(NettyServerBuilder serverBuilder) {
122160
// add a ratelimiter interceptor
123161
serverBuilder.intercept(this.rateLimiterInterceptor);
124162

@@ -134,7 +172,7 @@ protected void addInterceptor(NettyServerBuilder serverBuilder) {
134172
}
135173
}
136174

137-
protected void initServer(NettyServerBuilder serverBuilder) {
175+
protected final void initServer(NettyServerBuilder serverBuilder) {
138176
this.apiServer = serverBuilder.build();
139177
}
140178

Lines changed: 84 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,84 @@
1+
package org.tron.core.admission;
2+
3+
import io.grpc.ForwardingServerCallListener.SimpleForwardingServerCallListener;
4+
import io.grpc.Grpc;
5+
import io.grpc.Metadata;
6+
import io.grpc.ServerCall;
7+
import io.grpc.ServerCallHandler;
8+
import io.grpc.Status;
9+
import lombok.extern.slf4j.Slf4j;
10+
import org.tron.common.prometheus.MetricKeys;
11+
import org.tron.common.prometheus.Metrics;
12+
import org.tron.core.admission.ProtobufAdmissionException.Reason;
13+
14+
/**
15+
* Closes the call with {@code INVALID_ARGUMENT}, or {@code RESOURCE_EXHAUSTED} for a request over
16+
* the size limit, when {@link GuardedRequestMarshaller} hands over a rejection sentinel; the
17+
* service method is never invoked.
18+
*/
19+
@Slf4j(topic = "rpc")
20+
public final class AdmissionCheckedHandler<ReqT, RespT> implements ServerCallHandler<ReqT, RespT> {
21+
22+
private final ServerCallHandler<ReqT, RespT> delegate;
23+
private final GuardedRequestMarshaller<ReqT> marshaller;
24+
25+
AdmissionCheckedHandler(ServerCallHandler<ReqT, RespT> delegate,
26+
GuardedRequestMarshaller<ReqT> marshaller) {
27+
this.delegate = delegate;
28+
this.marshaller = marshaller;
29+
}
30+
31+
public ServerCallHandler<ReqT, RespT> getDelegate() {
32+
return delegate;
33+
}
34+
35+
public GuardedRequestMarshaller<ReqT> getMarshaller() {
36+
return marshaller;
37+
}
38+
39+
@Override
40+
public ServerCall.Listener<ReqT> startCall(ServerCall<ReqT, RespT> call, Metadata headers) {
41+
ServerCall.Listener<ReqT> next = delegate.startCall(call, headers);
42+
return new SimpleForwardingServerCallListener<ReqT>(next) {
43+
// Listener callbacks of one call are serialized by grpc.
44+
private boolean rejected;
45+
46+
@Override
47+
public void onMessage(ReqT message) {
48+
if (rejected) {
49+
return;
50+
}
51+
Reason reason = marshaller.rejectionReason(message);
52+
if (reason != null) {
53+
rejected = true;
54+
String method = call.getMethodDescriptor().getFullMethodName();
55+
if (logger.isDebugEnabled()) {
56+
logger.debug("gRPC {} from {} rejected by admission check: {}", method,
57+
call.getAttributes().get(Grpc.TRANSPORT_ATTR_REMOTE_ADDR), reason);
58+
}
59+
Metrics.counterInc(MetricKeys.Counter.GRPC_ADMISSION_REJECT, 1, method, reason.name());
60+
// The same status grpc gives an uncompressed message over the size limit.
61+
Status status = reason == Reason.TOO_LARGE
62+
? Status.RESOURCE_EXHAUSTED : Status.INVALID_ARGUMENT;
63+
call.close(status.withDescription(marshaller.rejectionMessage()), new Metadata());
64+
return;
65+
}
66+
super.onMessage(message);
67+
}
68+
69+
@Override
70+
public void onHalfClose() {
71+
if (!rejected) {
72+
super.onHalfClose();
73+
}
74+
}
75+
76+
@Override
77+
public void onReady() {
78+
if (!rejected) {
79+
super.onReady();
80+
}
81+
}
82+
};
83+
}
84+
}

0 commit comments

Comments
 (0)