DataNode does not autoscale under bulk-insert — HPA on CPU/mem never fires; per-task latency grows ~5× with input size

[saikishore143-hub]

On a multi-DataNode Milvus 2.5 cluster, bulk-insert throughput does not scale with input size because DataNode replicas remain pinned at HPA minReplicas for the entire run. HPA is on CPU + memory (Helm defaults), and the DataNode
import path is dominated by object-store IO and segment flush CPU and memory never cross the HPA threshold, so the pod count never grows.

IndexNode, by contrast, autoscales cleanly (up to 218 replicas in our largest run) because indexing is CPU-bound and the shipped HPA metrics track it.

Net effect: DataNode per-task latency grows ~5.4× as concurrent input grows 50×, while IndexNode per-task latency stays roughly flat (~1.4×).

We're looking for the recommended autoscaling signal for DataNode under bulk-insert workloads and also to undertand why the latencies are higher though the CPU and mem is in control for data nodes.

• DataNode max latency: 316 s → 1700 s as total ingest grows 1 M → 50 M rows (4 → 200 RPC calls of 250K, ~5.4× per-task latency).
• IndexNode max latency: 288 s → 417 s across the same range (~1.4×).
• IndexNode scaled to 218 replicas on the 50 M run.
• DataNode stayed at HPA minReplicas on every run.

[xiaofan-luan]

I would recommmend to uprade to latest 2.6.
Many of the problems might already be solved there.