milvus operator部署的milvus cluster etcd节点所在机器磁盘故障，使用了openlocal的本地卷方式，如何迁移etcd节点？

[bozai111]

帮忙分析下，milvus operator部署的milvus cluster，etcd-0节点所在机器磁盘故障，使用的openlocal本地卷，现在需要迁移实例到其他机器上。

操作步骤如下
1、故障的etcd节点被手动remove掉了，开始操作前先把故障的etcd节点加回来

#etcdctl --endpoints=10.xx.xx.xx:2379 member add etcd-0 --peer-urls=xx.xx.xx.xx:2380
Member 4d6b7595e8aed55b added to cluster 8fbb4727ea909544

ETCD_NAME="etcd-0"
ETCD_INITIAL_CLUSTER="etcd-2=xx.xx.xx.xx:2380,etcd-1=xx.xx.xx.xx.:2380,etcd-0=xx.xx.xx.xx.:2380"
ETCD_INITIAL_ADVERTISE_PEER_URLS="xx.xx.xx.xx:2380"
ETCD_INITIAL_CLUSTER_STATE="existing"

2、kubectl delete pvc data-etcd-0 -n milvus01
kubectl delete pvc, 卡在删除中，编辑pvc干掉 finalizer

3、删除pod
kubectl delete pod etcd-0 -n milvus01

每次delete pvc，都重新调度选了新的机器，新的PVC。不应该存在数据残留问题。

最后etcd节点状态还是不正常，处于complete的状态，不断重启，日志如下：

[bozai111]

etcd 07:08:48.41

etcd 07:08:48.41 Welcome to the Bitnami etcd container

etcd 07:08:48.41 Subscribe to project updates by watching https://github.com/bitnami/containers

etcd 07:08:48.41 Submit issues and feature requests at https://github.com/bitnami/containers/issues

etcd 07:08:48.41

etcd 07:08:48.41 INFO ==> ** Starting etcd setup **

etcd 07:08:48.42 INFO ==> Validating settings in ETCD_* env vars..

etcd 07:08:48.42 WARN ==> You set the environment variable ALLOW_NONE_AUTHENTICATION=yes. For safety reasons, do not use this flag in a production environment.

etcd 07:08:48.42 INFO ==> Initializing etcd

etcd 07:08:48.42 INFO ==> Generating etcd config file using env variables

etcd 07:08:53.47 INFO ==> xx.xx.xx.xx.:2380,http://xx.xx.xx.xx:2380

etcd 07:08:53.47 INFO ==> get_etcd_active_endpoints: xx.xx.xx.xx.:2380,http://xx.xx.xx.xx:2380

etcd 07:08:53.47 INFO ==> Detected data from previous deployments

etcd 07:08:53.47 INFO ==> The member will try to join the cluster by it's own

etcd 07:08:53.48 INFO ==> ** etcd setup finished! **

etcd 07:08:53.48 INFO ==> ** Starting etcd **

[bozai111]

etcd 07:08:53.48 INFO ==> ** Starting etcd **

{"level":"info","ts":"2026-08-21T07:08:53.497631Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_ADVERTISE_CLIENT_URLS","variable-value":"xx.xx.xx.xx:2379"}

{"level":"info","ts":"2026-08-21T07:08:53.497682Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_AUTO_COMPACTION_MODE","variable-value":"revision"}

{"level":"info","ts":"2026-08-21T07:08:53.497686Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_AUTO_COMPACTION_RETENTION","variable-value":"1000"}

{"level":"info","ts":"2026-08-21T07:08:53.497696Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_AUTO_TLS","variable-value":"false"}

{"level":"info","ts":"2026-08-21T07:08:53.497704Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_CLIENT_CERT_AUTH","variable-value":"false"}

{"level":"info","ts":"2026-08-21T07:08:53.497709Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_DATA_DIR","variable-value":"/bitnami/etcd/data"}

{"level":"info","ts":"2026-08-21T07:08:53.497714Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_ELECTION_TIMEOUT","variable-value":"2500"}

{"level":"info","ts":"2026-08-21T07:08:53.497729Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_HEARTBEAT_INTERVAL","variable-value":"500"}

{"level":"info","ts":"2026-08-21T07:08:53.497737Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_INITIAL_ADVERTISE_PEER_URLS","variable-value":"xx.xx.xx.xx:2380"}

{"level":"info","ts":"2026-08-21T07:08:53.497740Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_INITIAL_CLUSTER","variable-value":"etcd-0=xx.xx.xx.xx:2380,etcd-1=xx.xx.xx.xx.:2380,etcd-2=http://xx.xx.xx.xx:2380"}

{"level":"info","ts":"2026-08-21T07:08:53.497746Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_INITIAL_CLUSTER_STATE","variable-value":"new"}

{"level":"info","ts":"2026-08-21T07:08:53.497749Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_INITIAL_CLUSTER_TOKEN","variable-value":"etcd-cluster-k8s"}

{"level":"info","ts":"2026-08-21T07:08:53.497756Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_LISTEN_CLIENT_URLS","variable-value":"http://0.0.0.0:2379"}

{"level":"info","ts":"2026-08-21T07:08:53.497763Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_LISTEN_PEER_URLS","variable-value":"http://0.0.0.0:2380"}

{"level":"info","ts":"2026-08-21T07:08:53.497766Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_LOG_LEVEL","variable-value":"info"}

{"level":"info","ts":"2026-08-21T07:08:53.497772Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_NAME","variable-value":"etcd-0"}

{"level":"info","ts":"2026-08-21T07:08:53.497777Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_PEER_AUTO_TLS","variable-value":"false"}

{"level":"info","ts":"2026-08-21T07:08:53.497786Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_QUOTA_BACKEND_BYTES","variable-value":"4294967296"}

{"level":"warn","ts":"2026-08-21T07:08:53.497796Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_TRUSTED_CA_FILE="}

{"level":"warn","ts":"2026-08-21T07:08:53.497806Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_CONF_FILE=/opt/bitnami/etcd/conf/etcd.yaml"}

{"level":"warn","ts":"2026-08-21T07:08:53.497810Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_ON_K8S=yes"}

[bozai111]

{"level":"warn","ts":"2026-08-21T07:08:53.497814Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_SNAPSHOTS_DIR=/snapshots"}

{"level":"warn","ts":"2026-08-21T07:08:53.497818Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_BIN_DIR=/opt/bitnami/etcd/bin"}

{"level":"warn","ts":"2026-08-21T07:08:53.497822Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_VOLUME_DIR=/bitnami/etcd"}

{"level":"warn","ts":"2026-08-21T07:08:53.497826Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_CLUSTER_DOMAIN=xx.xx.xx.xx."}

{"level":"warn","ts":"2026-08-21T07:08:53.497829Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_DISASTER_RECOVERY=no"}

{"level":"warn","ts":"2026-08-21T07:08:53.497831Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_KEY_FILE="}

{"level":"warn","ts":"2026-08-21T07:08:53.497835Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_CONF_DIR=/opt/bitnami/etcd/conf"}

{"level":"warn","ts":"2026-08-21T07:08:53.497838Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_DAEMON_GROUP=etcd"}

{"level":"warn","ts":"2026-08-21T07:08:53.497841Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_START_FROM_SNAPSHOT=no"}

{"level":"warn","ts":"2026-08-21T07:08:53.497844Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_INIT_SNAPSHOT_FILENAME="}

{"level":"warn","ts":"2026-08-21T07:08:53.497853Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_INIT_SNAPSHOTS_DIR=/init-snapshot"}

{"level":"warn","ts":"2026-08-21T07:08:53.497859Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_TMP_DIR=/opt/bitnami/etcd/tmp"}

{"level":"warn","ts":"2026-08-21T07:08:53.497863Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_BASE_DIR=/opt/bitnami/etcd"}

{"level":"warn","ts":"2026-08-21T07:08:53.497866Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_CERT_FILE="}

{"level":"warn","ts":"2026-08-21T07:08:53.497869Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_NEW_MEMBERS_ENV_FILE=/bitnami/etcd/data/new_member_envs"}

{"level":"warn","ts":"2026-08-21T07:08:53.497871Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_DAEMON_USER=etcd"}

{"level":"warn","ts":"2026-08-21T07:08:53.498043Z","caller":"embed/config.go:689","msg":"Running http and grpc server on single port. This is not recommended for production."}

{"level":"info","ts":"2026-08-21T07:08:53.498521Z","caller":"etcdmain/etcd.go:73","msg":"Running: ","args":["etcd"]}

{"level":"info","ts":"2026-08-21T07:08:53.498581Z","caller":"etcdmain/etcd.go:116","msg":"server has been already initialized","data-dir":"/bitnami/etcd/data","dir-type":"member"}

{"level":"warn","ts":"2026-08-21T07:08:53.498597Z","caller":"embed/config.go:689","msg":"Running http and grpc server on single port. This is not recommended for production."}

{"level":"info","ts":"2026-08-21T07:08:53.498604Z","caller":"embed/etcd.go:128","msg":"configuring peer listeners","listen-peer-urls":["http://0.0.0.0:2380"]}

{"level":"info","ts":"2026-08-21T07:08:53.498686Z","caller":"embed/etcd.go:136","msg":"configuring client listeners","listen-client-urls":["http://0.0.0.0:2379"]}

{"level":"info","ts":"2026-08-21T07:08:53.498749Z","caller":"embed/etcd.go:311","msg":"starting an etcd server","etcd-version":"3.5.16","git-sha":"f20bbad","go-version":"go1.22.7","go-os":"linux","go-arch":"amd64","max-cpu-set":48,"max-cpu-available":48,"member-initialized":true,"name":"etcd-0","data-dir":"/bitnami/etcd/data","wal-dir":"","wal-dir-dedicated":"","member-dir":"/bitnami/etcd/data/member","force-new-cluster":false,"heartbeat-interval":"500ms","election-timeout":"2.5s","initial-election-tick-advance":true,"snapshot-count":100000,"max-wals":5,"max-snapshots":5,"snapshot-catchup-entries":5000,"initial-advertise-peer-urls":["xx.xx.xx.xx:2380"],"listen-peer-urls":["http://0.0.0.0:2380"],"advertise-client-urls":["xx.xx.xx.xx:2379"],"listen-client-urls":["http://0.0.0.0:2379"],"listen-metrics-urls":[],"cors":[""],"host-whitelist":[""],"initial-cluster":"","initial-cluster-state":"new","initial-cluster-token":"","quota-backend-bytes":4294967296,"max-request-bytes":1572864,"max-concurrent-streams":4294967295,"pre-vote":true,"initial-corrupt-check":false,"corrupt-check-time-interval":"0s","compact-check-time-enabled":false,"compact-check-time-interval":"1m0s","auto-compaction-mode":"revision","auto-compaction-retention":"1µs","auto-compaction-interval":"1µs","discovery-url":"","discovery-proxy":"","downgrade-check-interval":"5s"}

{"level":"info","ts":"2026-08-21T07:08:53.503756Z","caller":"etcdserver/backend.go:81","msg":"opened backend db","path":"/bitnami/etcd/data/member/snap/db","took":"4.895771ms"}

{"level":"info","ts":"2026-08-21T07:08:53.504085Z","caller":"etcdserver/server.go:532","msg":"No snapshot found. Recovering WAL from scratch!"}

[bozai111]

{"level":"info","ts":"2026-08-21T07:08:53.504268Z","caller":"etcdserver/raft.go:540","msg":"restarting local member","cluster-id":"8fbb4727ea909544","local-member-id":"cd1ccfec99c10bbf","commit-index":3}

{"level":"info","ts":"2026-08-21T07:08:53.504297Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"cd1ccfec99c10bbf switched to configuration voters=()"}

{"level":"info","ts":"2026-08-21T07:08:53.504323Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"cd1ccfec99c10bbf became follower at term 1"}

{"level":"info","ts":"2026-08-21T07:08:53.504331Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"newRaft cd1ccfec99c10bbf [peers: [], term: 1, commit: 3, applied: 0, lastindex: 3, lastterm: 1]"}

{"level":"warn","ts":"2026-08-21T07:08:53.504445Z","caller":"auth/store.go:1241","msg":"simple token is not cryptographically signed"}

{"level":"info","ts":"2026-08-21T07:08:53.505124Z","caller":"mvcc/kvstore.go:423","msg":"kvstore restored","current-rev":1}

{"level":"info","ts":"2026-08-21T07:08:53.505409Z","caller":"etcdserver/quota.go:117","msg":"enabled backend quota","quota-name":"v3-applier","quota-size-bytes":4294967296,"quota-size":"4.3 GB"}

{"level":"info","ts":"2026-08-21T07:08:53.505630Z","caller":"etcdserver/server.go:873","msg":"starting etcd server","local-member-id":"cd1ccfec99c10bbf","local-server-version":"3.5.16","cluster-version":"to_be_decided"}

{"level":"info","ts":"2026-08-21T07:08:53.505674Z","caller":"etcdserver/server.go:773","msg":"starting initial election tick advance","election-ticks":5}

{"level":"info","ts":"2026-08-21T07:08:53.505689Z","caller":"fileutil/purge.go:50","msg":"started to purge file","dir":"/bitnami/etcd/data/member/snap","suffix":"snap.db","max":5,"interval":"30s"}

{"level":"info","ts":"2026-08-21T07:08:53.505804Z","caller":"fileutil/purge.go:50","msg":"started to purge file","dir":"/bitnami/etcd/data/member/snap","suffix":"snap","max":5,"interval":"30s"}

{"level":"info","ts":"2026-08-21T07:08:53.505812Z","caller":"fileutil/purge.go:50","msg":"started to purge file","dir":"/bitnami/etcd/data/member/wal","suffix":"wal","max":5,"interval":"30s"}

{"level":"info","ts":"2026-08-21T07:08:53.505857Z","caller":"v3rpc/health.go:61","msg":"grpc service status changed","service":"","status":"SERVING"}

{"level":"info","ts":"2026-08-21T07:08:53.506018Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"cd1ccfec99c10bbf switched to configuration voters=(2280349143424296786)"}

{"level":"info","ts":"2026-08-21T07:08:53.506095Z","caller":"membership/cluster.go:421","msg":"added member","cluster-id":"8fbb4727ea909544","local-member-id":"cd1ccfec99c10bbf","added-peer-id":"1fa56d76d1abe752","added-peer-peer-urls":["http://xx.xx.xx.xx:2380"]}

{"level":"info","ts":"2026-08-21T07:08:53.506110Z","caller":"rafthttp/peer.go:133","msg":"starting remote peer","remote-peer-id":"1fa56d76d1abe752"}

{"level":"info","ts":"2026-08-21T07:08:53.506133Z","caller":"rafthttp/pipeline.go:72","msg":"started HTTP pipelining with remote peer","local-member-id":"cd1ccfec99c10bbf","remote-peer-id":"1fa56d76d1abe752"}

{"level":"info","ts":"2026-08-21T07:08:53.506215Z","caller":"rafthttp/stream.go:169","msg":"started stream writer with remote peer","local-member-id":"cd1ccfec99c10bbf","remote-peer-id":"1fa56d76d1abe752"}

{"level":"info","ts":"2026-08-21T07:08:53.506290Z","caller":"rafthttp/peer.go:137","msg":"started remote peer","remote-peer-id":"1fa56d76d1abe752"}

[LoveEachDay]

@bozai111 Could you modify the env ETCD_INITIAL_CLUSTER_STATE from new to existing?

[bozai111]

Has this method been verified?

[bozai111]

这个参数的值是Milvus operator启动pod时，通过脚本注入的，如果直接修改，是否有作用？这个直接修改的方式有实践过么？