Fix stale IoTConsensus peers after restart during removal

[d-wang-commit]

Description

Restart during peer removal

A surviving IoTConsensus V1 replica can restart after the removed peer's sync channel has been deleted but before ConfigNode commits the updated replica set. Its startup response still includes the removed peer, so it recreates that peer's dispatcher. Previously, completing the metadata update did not correct this recovered membership. Failed replication to the removed peer could then pin the minimum sync index and keep syncLag above zero.

Reconcile committed membership

Commit the replica-set removal before entering the new RESET_IOT_CONSENSUS_V1_PEER_LIST state. For IoTConsensus V1 DataRegions, send the final peer list to every retained replica through the existing resetPeerList RPC, including temporarily unavailable replicas. The existing DataNode handler waits for consensus startup before applying the list.

Metadata failures and unsuccessful V1 resets perform an initial attempt and up to three retries per state, sleeping five seconds between retries. Exhaustion marks the procedure as failed. The business state is persisted by the existing procedure framework and can resume after ConfigNode recovery; the in-memory cycle counter restarts after recovery. Existing state IDs are unchanged, and the new state is appended. No DataNode production code or RPC definitions are changed.

Ratis and IoTConsensus V2 crash ITs now exclude the V1-only state from their expected kill points. V1 retains coverage of the new state, and the assertion requiring all expected kill points to trigger is preserved.

Tests

• UT added: RestartDuringRemovePeerTest covers metadata ordering, metadata failures for V1 and V2, missing or unsuccessful reset responses, exceptions, recovery, idempotent replay, retry exhaustion, interruption, and protocol gating. RemovePeerRestartConsensusTest covers stale membership, syncLag convergence, continued writes, and another restart using real IoTConsensus instances and loopback Thrift RPCs. StalePeerListRestartReproductionTest provides the stale-startup reproduction and an explicit reset control.
• IT modified: cnCrashTest in IoTDBRegionMigrateConfigNodeCrashForRatisIT, IoTDBRegionMigrateConfigNodeCrashIoTV2BatchIT, and IoTDBRegionMigrateConfigNodeCrashIoTV2StreamIT excludes the unreachable V1-only kill point.
• UT run: All 20 targeted tests passed across RestartDuringRemovePeerTest, RemovePeerRestartConsensusTest, StalePeerListRestartReproductionTest, RemoveRegionPeerProcedureTest, and STMProcedureTest. The CN-driven regression reduced syncLag from 8 to 0; further writes and another restart kept it at 0.
• IT status: Execution of the three modified cnCrashTest scenarios is pending; cluster validation is continuing.
• Build checks and boundaries: Spotless passed for the three updated IT files, and git diff --check passed. The consensus component tests use same-JVM restarts, an in-memory state machine, and a test internal-service facade; they do not launch full DataNode processes.

Side effects and risks

• If a retained replica cannot acknowledge the reset within all attempts, finalization fails and stale membership may remain uncorrected. No background reconciliation is added for exhausted or previously completed procedures.
• Retry sleeps occupy a procedure worker, adding up to 15 seconds of sleep per failing state in addition to RPC time.
• ConfigNodes recovering the new state must contain this change; older binaries cannot interpret its appended state ID.
• Existing DataNode dispatcher shutdown behavior is unchanged. The component regression verifies membership and syncLag accounting, not complete dispatcher-thread termination.

---

This PR has:

• been self-reviewed.
• added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.

---

Key changed/added classes (or packages if there are too many classes) in this PR

• RemoveRegionPeerProcedure: commit metadata, execute V1 reconciliation, and bound finalization retries.
• RegionMaintainHandler: return metadata status and verify reset responses from every retained V1 replica.
• RemoveRegionPeerState: append the V1 reconciliation state.
• ProcedureMessages (en/zh): add reconciliation and retry diagnostics.
• RestartDuringRemovePeerTest, RemovePeerRestartConsensusTest, StalePeerListRestartReproductionTest: cover restart, replication, and finalization behavior.
• Ratis and IoTConsensus V2 ConfigNode crash ITs: filter the V1-only kill point.