Skip to content

Commit abb9ef9

Browse files
VGalaxiescodex
andauthored
Subscription: add topic owner epoch fencing (#17780)
* Subscription: add topic owner epoch fencing * Implement alter topic owner entrypoints * Add topic owner update show test * Subscription: address topic owner review comments * Implement subscription topic owner lease renewal * Use duration-based subscription owner lease renewal * Renew subscription owner leases in ConfigNode heartbeat * Rework subscription topic owner epoch fencing and lease Land the technical-review corrections together: - Seek fencing in SubscriptionReceiverV1 before any position-mutating op. - Move the owner lease off the node heartbeat onto a dedicated, fine-grained SubscriptionOwnerLeaseSyncer pushing relative remaining durations. - Make the absolute lease expiry DataNode-local runtime state only: it is set solely from the lease heartbeat, never an attribute, never serialized, and not part of equals/hashCode. CN persists only owner-id, owner-epoch, max-owner-epoch and owner-lease-duration-ms. - Add globally-monotonic, never-reused owner epochs via max-owner-epoch that survives clearOwner. - Drain owner transfers via a ConfigNode admission-wait (blockSince + L + H) instead of CN-side lease-not-expired rejection/retry. - Remove the dead absolute-timestamp lease machinery (renewOwnerLease*, getOwnerLeaseRemainingTimeMs, isActiveOwnerLease/isOwnerTransfer, the owner-lease-expire-time-ms attribute key, the alterTopic retry loop) and update the unit tests accordingly. * Move subscription owner fencing messages into i18n constants Address the i18n review comments: the owner-validation exception messages in TopicMeta now use SubscriptionMessages (en/zh), and the alter-topic warn in SubscriptionCoordinator uses ManagerMessages. The previously flagged renewTopicOwnerLease message no longer exists after the lease rework. * Reword owner-lease syncer TODO to satisfy the todo-check CI gate * Enable the local topic-owner fencing IT (drop stale @ignore) AbstractSubscriptionLocalIT already enables subscription, so the end-to-end stale-owner-fencing + current-owner-consume scenario runs; verified locally via mvn verify -Dit.test=IoTDBSubscriptionTopicOwnerIT (tests=2, failures=0, errors=0, skipped=0). * Rename SN owner identifiers to neutral owner names in tests Drop 'SN' wording from the PR's new code: owner-id values sn1/sn2 -> owner1/owner2, consumer ids and vars stale_sn/current_sn/oldSnConsumer/newSnConsumer -> owner-based names, and the SN/DN/CN comment -> owner/DataNode/ConfigNode. Pure rename; all affected unit tests still pass. * Add configurable minimum owner-lease-duration floor (default 1 min) Reject owner-lease-duration-ms below a configurable floor at topic create/alter admission, so the lease stays well above the ~5s renewal heartbeat and a missed beat cannot falsely fence a healthy owner. - New config subscription_owner_lease_duration_ms_min (CommonConfig, default 60000), loaded by CommonDescriptor, exposed via SubscriptionConfig. - Enforced in SubscriptionInfo.validateTopicConfig (create + alter admission), with an en/zh ManagerMessages constant. Deserialize stays lenient (>0) so already-persisted topics still load if the floor is later raised. - IT framework: setSubscriptionOwnerLeaseDurationMsMin on the CommonConfig interface + Mpp/MppShared/Remote impls; the two owner ITs lower the floor in setUp to keep their short leases (and stay fast). Verified the local IT passes (tests=2, failures=0) end-to-end. * Clarify owner-lease drain wait depends on the heartbeat interval too Address review feedback: the ConfigNode admission wait is (lease duration + one renewal/heartbeat interval) measured on its own clock, not the lease duration alone, since a renewal may have been in flight to a DataNode just before renewal was stopped. Updated the drain test comment to reflect this; the >= lease-duration lower bound still holds because the actual wait is duration + one interval. --------- Co-authored-by: OpenAI Codex <codex@openai.com>
1 parent aec7dfb commit abb9ef9

80 files changed

Lines changed: 2889 additions & 24 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎integration-test/src/main/java/org/apache/iotdb/it/env/cluster/config/MppCommonConfig.java‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -498,6 +498,21 @@ public CommonConfig setIsPipeEnableMemoryCheck(boolean isPipeEnableMemoryCheck)
498498
return this;
499499
}
500500

501+
@Override
502+
public CommonConfig setSubscriptionEnabled(boolean subscriptionEnabled) {
503+
setProperty("subscription_enabled", String.valueOf(subscriptionEnabled));
504+
return this;
505+
}
506+
507+
@Override
508+
public CommonConfig setSubscriptionOwnerLeaseDurationMsMin(
509+
long subscriptionOwnerLeaseDurationMsMin) {
510+
setProperty(
511+
"subscription_owner_lease_duration_ms_min",
512+
String.valueOf(subscriptionOwnerLeaseDurationMsMin));
513+
return this;
514+
}
515+
501516
@Override
502517
public CommonConfig setPipeAirGapReceiverEnabled(boolean isPipeAirGapReceiverEnabled) {
503518
setProperty("pipe_air_gap_receiver_enabled", String.valueOf(isPipeAirGapReceiverEnabled));

‎integration-test/src/main/java/org/apache/iotdb/it/env/cluster/config/MppSharedCommonConfig.java‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -514,6 +514,21 @@ public CommonConfig setIsPipeEnableMemoryCheck(boolean isPipeEnableMemoryCheck)
514514
return this;
515515
}
516516

517+
@Override
518+
public CommonConfig setSubscriptionEnabled(boolean subscriptionEnabled) {
519+
dnConfig.setSubscriptionEnabled(subscriptionEnabled);
520+
cnConfig.setSubscriptionEnabled(subscriptionEnabled);
521+
return this;
522+
}
523+
524+
@Override
525+
public CommonConfig setSubscriptionOwnerLeaseDurationMsMin(
526+
long subscriptionOwnerLeaseDurationMsMin) {
527+
dnConfig.setSubscriptionOwnerLeaseDurationMsMin(subscriptionOwnerLeaseDurationMsMin);
528+
cnConfig.setSubscriptionOwnerLeaseDurationMsMin(subscriptionOwnerLeaseDurationMsMin);
529+
return this;
530+
}
531+
517532
@Override
518533
public CommonConfig setPipeAirGapReceiverEnabled(boolean isPipeAirGapReceiverEnabled) {
519534
dnConfig.setPipeAirGapReceiverEnabled(isPipeAirGapReceiverEnabled);

‎integration-test/src/main/java/org/apache/iotdb/it/env/remote/config/RemoteCommonConfig.java‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -363,6 +363,17 @@ public CommonConfig setIsPipeEnableMemoryCheck(boolean isPipeEnableMemoryCheck)
363363
return this;
364364
}
365365

366+
@Override
367+
public CommonConfig setSubscriptionEnabled(boolean subscriptionEnabled) {
368+
return this;
369+
}
370+
371+
@Override
372+
public CommonConfig setSubscriptionOwnerLeaseDurationMsMin(
373+
long subscriptionOwnerLeaseDurationMsMin) {
374+
return this;
375+
}
376+
366377
@Override
367378
public CommonConfig setPipeAirGapReceiverEnabled(boolean isPipeAirGapReceiverEnabled) {
368379
return this;

‎integration-test/src/main/java/org/apache/iotdb/itbase/env/CommonConfig.java‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -160,6 +160,10 @@ CommonConfig setEnableAutoLeaderBalanceForIoTConsensus(
160160

161161
CommonConfig setIsPipeEnableMemoryCheck(boolean isPipeEnableMemoryCheck);
162162

163+
CommonConfig setSubscriptionEnabled(boolean subscriptionEnabled);
164+
165+
CommonConfig setSubscriptionOwnerLeaseDurationMsMin(long subscriptionOwnerLeaseDurationMsMin);
166+
163167
CommonConfig setPipeAirGapReceiverEnabled(boolean isPipeAirGapReceiverEnabled);
164168

165169
CommonConfig setDriverTaskExecutionTimeSliceInMs(long driverTaskExecutionTimeSliceInMs);
Lines changed: 267 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,267 @@
1+
/*
2+
* Licensed to the Apache Software Foundation (ASF) under one
3+
* or more contributor license agreements. See the NOTICE file
4+
* distributed with this work for additional information
5+
* regarding copyright ownership. The ASF licenses this file
6+
* to you under the Apache License, Version 2.0 (the
7+
* "License"); you may not use this file except in compliance
8+
* with the License. You may obtain a copy of the License at
9+
*
10+
* http://www.apache.org/licenses/LICENSE-2.0
11+
*
12+
* Unless required by applicable law or agreed to in writing,
13+
* software distributed under the License is distributed on an
14+
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15+
* KIND, either express or implied. See the License for the
16+
* specific language governing permissions and limitations
17+
* under the License.
18+
*/
19+
20+
package org.apache.iotdb.subscription.it.cluster;
21+
22+
import org.apache.iotdb.isession.ISession;
23+
import org.apache.iotdb.it.env.EnvFactory;
24+
import org.apache.iotdb.it.framework.IoTDBTestRunner;
25+
import org.apache.iotdb.itbase.category.ClusterIT;
26+
import org.apache.iotdb.rpc.subscription.config.TopicConstant;
27+
import org.apache.iotdb.rpc.subscription.exception.SubscriptionOwnerFencedException;
28+
import org.apache.iotdb.session.subscription.SubscriptionTreeSession;
29+
import org.apache.iotdb.session.subscription.consumer.tree.SubscriptionTreePullConsumer;
30+
import org.apache.iotdb.session.subscription.model.Topic;
31+
import org.apache.iotdb.subscription.it.AbstractSubscriptionIT;
32+
import org.apache.iotdb.subscription.it.IoTDBSubscriptionITConstant;
33+
34+
import org.junit.After;
35+
import org.junit.Assert;
36+
import org.junit.Before;
37+
import org.junit.Test;
38+
import org.junit.experimental.categories.Category;
39+
import org.junit.runner.RunWith;
40+
41+
import java.util.Properties;
42+
43+
/**
44+
* Owner-fencing correctness under owner/DataNode/ConfigNode partition topologies, simulated at the
45+
* node-process level via the IT framework (shutdown/start of ConfigNodes and DataNodes; ConfigNode
46+
* failover via leader shutdown). Runs on a 3 ConfigNode + 3 DataNode cluster.
47+
*
48+
* <p>Covers:
49+
*
50+
* <ul>
51+
* <li>multi-DataNode: a stale owner is fenced for a topic whose data spans several DataNodes;
52+
* <li>ConfigNode failover: the owner epoch is persisted via consensus, so the new leader keeps
53+
* fencing the stale owner after the previous leader is shut down;
54+
* <li>DataNode restart: after a DataNode is shut down and restarted, the owner metadata is
55+
* re-synced and the stale owner stays fenced.
56+
* </ul>
57+
*/
58+
@RunWith(IoTDBTestRunner.class)
59+
@Category({ClusterIT.class})
60+
public class IoTDBSubscriptionTopicOwnerPartitionIT extends AbstractSubscriptionIT {
61+
62+
private static final String HEAD_PATH = "root.topic_owner_partition";
63+
64+
@Override
65+
@Before
66+
public void setUp() throws Exception {
67+
super.setUp();
68+
EnvFactory.getEnv()
69+
.getConfig()
70+
.getCommonConfig()
71+
.setSubscriptionEnabled(true)
72+
.setPipeMemoryManagementEnabled(false)
73+
.setIsPipeEnableMemoryCheck(false)
74+
// Lower the owner-lease floor so the test can use a short lease and stay fast.
75+
.setSubscriptionOwnerLeaseDurationMsMin(1000);
76+
// 3 ConfigNodes (for failover) + 3 DataNodes (for multi-DataNode data placement).
77+
EnvFactory.getEnv().initClusterEnvironment(3, 3);
78+
}
79+
80+
@Override
81+
@After
82+
public void tearDown() throws Exception {
83+
EnvFactory.getEnv().cleanClusterEnvironment();
84+
super.tearDown();
85+
}
86+
87+
@Test
88+
public void testStaleOwnerFencedOnMultiDataNodeTopic() throws Exception {
89+
final String host = EnvFactory.getEnv().getIP();
90+
final int port = Integer.parseInt(EnvFactory.getEnv().getPort());
91+
final String topicName = "topic_owner_multi_dn";
92+
93+
createOwnedTopic(host, port, topicName, "owner2", 6L);
94+
try {
95+
// Spread data across multiple devices (and thus, with multiple data regions, multiple
96+
// DataNodes) before consuming.
97+
insertDataAcrossDevices();
98+
99+
// A stale owner (epoch 5 < current 6) must be fenced regardless of which DataNode serves the
100+
// request.
101+
assertSubscribeFenced(host, port, topicName, "stale_multi_dn", "owner1", 5L);
102+
103+
// The current owner can subscribe.
104+
try (final SubscriptionTreePullConsumer currentOwnerConsumer =
105+
ownerConsumer(
106+
host, port, "current_multi_dn", "topic_owner_multi_dn_group", "owner2", 6L)) {
107+
currentOwnerConsumer.open();
108+
currentOwnerConsumer.subscribe(topicName);
109+
currentOwnerConsumer.unsubscribe(topicName);
110+
}
111+
} finally {
112+
dropTopic(host, port, topicName);
113+
}
114+
}
115+
116+
@Test
117+
public void testConfigNodeFailoverPersistsOwnerEpochAndKeepsFencing() throws Exception {
118+
final String host = EnvFactory.getEnv().getIP();
119+
final int port = Integer.parseInt(EnvFactory.getEnv().getPort());
120+
final String topicName = "topic_owner_cn_failover";
121+
final long ownerLeaseDurationMs = 2000L;
122+
123+
try (final SubscriptionTreeSession session = new SubscriptionTreeSession(host, port)) {
124+
session.open();
125+
final Properties properties = new Properties();
126+
properties.put(TopicConstant.PATH_KEY, HEAD_PATH + ".**");
127+
properties.put(TopicConstant.START_TIME_KEY, "0");
128+
properties.put(TopicConstant.OWNER_ID_KEY, "owner1");
129+
properties.put(TopicConstant.OWNER_EPOCH_KEY, "5");
130+
properties.put(
131+
TopicConstant.OWNER_LEASE_DURATION_MS_KEY, String.valueOf(ownerLeaseDurationMs));
132+
session.createTopic(topicName, properties);
133+
134+
// Transfer to owner2/epoch 6 (this persists owner + epoch via ConfigNode consensus).
135+
session.alterTopicOwner(topicName, "owner2", 6L);
136+
Assert.assertTrue(getTopicAttributes(session, topicName).contains("owner-epoch=6"));
137+
}
138+
139+
// Fail over: shut down the current leader ConfigNode.
140+
final int leaderIndex = EnvFactory.getEnv().getLeaderConfigNodeIndex();
141+
EnvFactory.getEnv().shutdownConfigNode(leaderIndex);
142+
143+
try {
144+
// After failover, the new leader must still report the persisted owner/epoch (consensus), and
145+
// a stale owner (epoch 5) must remain fenced.
146+
IoTDBSubscriptionITConstant.AWAIT.untilAsserted(
147+
() -> {
148+
try (final SubscriptionTreeSession session = new SubscriptionTreeSession(host, port)) {
149+
session.open();
150+
final String attributes = getTopicAttributes(session, topicName);
151+
Assert.assertTrue(attributes.contains("owner-id=owner2"));
152+
Assert.assertTrue(attributes.contains("owner-epoch=6"));
153+
}
154+
});
155+
assertSubscribeFenced(host, port, topicName, "stale_after_failover", "owner1", 5L);
156+
} finally {
157+
EnvFactory.getEnv().startConfigNode(leaderIndex);
158+
dropTopic(host, port, topicName);
159+
}
160+
}
161+
162+
@Test
163+
public void testDataNodeRestartReSyncsOwnerFencing() throws Exception {
164+
final String host = EnvFactory.getEnv().getIP();
165+
final int port = Integer.parseInt(EnvFactory.getEnv().getPort());
166+
final String topicName = "topic_owner_dn_restart";
167+
168+
createOwnedTopic(host, port, topicName, "owner2", 6L);
169+
try {
170+
// Restart a DataNode: it loses in-memory owner metadata and must re-sync it on restart.
171+
EnvFactory.getEnv().shutdownDataNode(1);
172+
EnvFactory.getEnv().startDataNode(1);
173+
174+
// After the DataNode comes back and re-syncs owner metadata, the stale owner stays fenced.
175+
IoTDBSubscriptionITConstant.AWAIT.untilAsserted(
176+
() ->
177+
assertSubscribeFenced(host, port, topicName, "stale_after_dn_restart", "owner1", 5L));
178+
} finally {
179+
dropTopic(host, port, topicName);
180+
}
181+
}
182+
183+
private static void createOwnedTopic(
184+
final String host,
185+
final int port,
186+
final String topicName,
187+
final String ownerId,
188+
final long ownerEpoch)
189+
throws Exception {
190+
try (final SubscriptionTreeSession session = new SubscriptionTreeSession(host, port)) {
191+
session.open();
192+
final Properties properties = new Properties();
193+
properties.put(TopicConstant.PATH_KEY, HEAD_PATH + ".**");
194+
properties.put(TopicConstant.START_TIME_KEY, "0");
195+
properties.put(TopicConstant.OWNER_ID_KEY, ownerId);
196+
properties.put(TopicConstant.OWNER_EPOCH_KEY, String.valueOf(ownerEpoch));
197+
session.createTopic(topicName, properties);
198+
}
199+
}
200+
201+
private static void dropTopic(final String host, final int port, final String topicName) {
202+
try (final SubscriptionTreeSession session = new SubscriptionTreeSession(host, port)) {
203+
session.open();
204+
session.dropTopicIfExists(topicName);
205+
} catch (final Exception e) {
206+
// best-effort cleanup
207+
}
208+
}
209+
210+
private static SubscriptionTreePullConsumer ownerConsumer(
211+
final String host,
212+
final int port,
213+
final String consumerId,
214+
final String consumerGroupId,
215+
final String ownerId,
216+
final long ownerEpoch) {
217+
return new SubscriptionTreePullConsumer.Builder()
218+
.host(host)
219+
.port(port)
220+
.consumerId(consumerId)
221+
.consumerGroupId(consumerGroupId)
222+
.ownerId(ownerId)
223+
.ownerEpoch(ownerEpoch)
224+
.autoCommit(false)
225+
.buildPullConsumer();
226+
}
227+
228+
private static void assertSubscribeFenced(
229+
final String host,
230+
final int port,
231+
final String topicName,
232+
final String consumerId,
233+
final String ownerId,
234+
final long ownerEpoch)
235+
throws Exception {
236+
try (final SubscriptionTreePullConsumer staleOwnerConsumer =
237+
ownerConsumer(host, port, consumerId, topicName + "_stale_group", ownerId, ownerEpoch)) {
238+
staleOwnerConsumer.open();
239+
Assert.assertThrows(
240+
SubscriptionOwnerFencedException.class, () -> staleOwnerConsumer.subscribe(topicName));
241+
}
242+
}
243+
244+
private void insertDataAcrossDevices() throws Exception {
245+
try (final ISession session = EnvFactory.getEnv().getSessionConnection()) {
246+
for (int device = 0; device < 6; device++) {
247+
for (int i = 0; i < 10; i++) {
248+
session.executeNonQueryStatement(
249+
String.format(
250+
"insert into %s.d%s(time, s1) values (%s, %s)", HEAD_PATH, device, i, i));
251+
}
252+
}
253+
session.executeNonQueryStatement("flush");
254+
}
255+
}
256+
257+
private static String getTopicAttributes(
258+
final SubscriptionTreeSession session, final String topicName) throws Exception {
259+
for (final Topic topic : session.getTopics()) {
260+
if (topicName.equals(topic.getTopicName())) {
261+
return topic.getTopicAttributes();
262+
}
263+
}
264+
Assert.fail("Topic " + topicName + " should exist.");
265+
return "";
266+
}
267+
}

‎integration-test/src/test/java/org/apache/iotdb/subscription/it/local/AbstractSubscriptionLocalIT.java‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,7 @@ public void setUp() throws Exception {
3636
EnvFactory.getEnv()
3737
.getConfig()
3838
.getCommonConfig()
39+
.setSubscriptionEnabled(true)
3940
.setPipeMemoryManagementEnabled(false)
4041
.setIsPipeEnableMemoryCheck(false);
4142

0 commit comments

Comments
 (0)