fix(loader): degrade unsupported fastsafetensors capabilities

Author: ABNER-1

docs/backend/server_arguments.md

@@ -205,34 +205,87 @@ The following legacy options and environment variables were removed and must no
 
 ### FastSafeTensors loader configuration
 
-When `LOAD_METHOD=fastsafetensors`, RTP-LLM uses the config-driven `AutoLoader`.
+When `LOAD_METHOD=fastsafetensors`, or when the default `auto` mode selects the
+FastSafeTensors path, RTP-LLM uses the config-driven `AutoLoader`. RTP checks
+the installed package capabilities before loading: the full capability set uses
+bounded `per-expert` delivery, a package without `dim0_split_templates` falls
+back to the higher-memory `full-stacked` compatibility path, and a package
+without `local_copyout_filter` continues with full materialization and RTP
+consumer-side filtering. A missing package/`AutoLoader`, an import/ABI failure,
+an unmet AUTO prerequisite, or an insufficient memory preflight falls back to
+`scratch`. These compatibility paths apply to both `auto` and an explicit
+`LOAD_METHOD=fastsafetensors`, so package age alone does not fail model startup.
+The two optional keywords control independent optimizations:
+
+| Capability | Present | Missing |
+|---|---|---|
+| `local_copyout_filter` | rank-local copy-out | full materialization, RTP consumer filtering |
+| `dim0_split_templates` | bounded `per-expert` MoE delivery | `full-stacked` MoE delivery |
+
+When a degraded FastSafeTensors mode remains usable, RTP logs
+`requested_mode`, `effective_mode` and `degraded_reason`. A scratch fallback
+contains `falls back to scratch`; package absence is INFO and other fallback
+causes are WARNING. CI or image builds that require both optimizations must
+install the matching wheel and treat a missing capability as a packaging
+failure. Set `RTP_LLM_EXPECT_FASTSAFETENSORS_TIER=per-expert` for the installed
+wheel contract test to turn a lower tier into a test failure; supported tiers
+are `scratch`, `consumer-filter`, `full-stacked`, and `per-expert`.
+
 Pass the standard fastsafetensors configuration as either an inline JSON string
-or a JSON file path. Inline JSON has higher priority when both are set:
+or a JSON file path. The installed FastSafeTensors version defines the precise
+configuration defaults and precedence:
 
 ```bash
-# Inline JSON string
-export FASTSAFETENSORS_CONFIG_JSON='{"loader":"base","base":{"copier_type":"nogds"}}'
+# Inline JSON string; progress is controlled by the upstream parallel config.
+export FASTSAFETENSORS_CONFIG_JSON='{"loader":"base","base":{"copier_type":"nogds"},"parallel":{"use_tqdm_on_load":true}}'
 
 # JSON file path; the file contains the same JSON object
 export FASTSAFETENSORS_CONFIG=/path/to/fastsafetensors.json
 ```
 
+The same configuration also affects `auto` selection. RTP reads
+`estimated_peak_device_bytes` from the installed package; missing or invalid
+values use the historical `3 × max checkpoint shard` estimate. Larger buffers,
+queues or producer counts can raise `transient_mem` enough for `auto` to choose
+`scratch`. Inspect the `fastsafetensor memory check` log and its `enough` field.
+
 For compatibility with existing development environments,
-`FASTSAFETENSORS_NOGDS=1` remains supported. Before constructing `AutoLoader`,
-RTP-LLM directly overrides `FASTSAFETENSORS_CONFIG_JSON` with
+`FASTSAFETENSORS_NOGDS=1` remains supported. Before memory preflight or
+constructing `AutoLoader`, RTP-LLM overrides `FASTSAFETENSORS_CONFIG_JSON`
+process-wide with
 `{"loader":"base","base":{"copier_type":"nogds"}}`. This compatibility switch
-therefore takes priority over other fastsafetensors configuration. Prefer one
-of the standard configuration variables above for new deployments.
+therefore remains in effect for subsequent loaders in the same process. Prefer
+one of the standard configuration variables above for new deployments. When
+`FASTSAFETENSORS_CONFIG` is also set, the final precedence remains an upstream
+package contract; current pinned wheels prefer the inline JSON value.
 
 Stacked MoE checkpoints use bounded-memory per-expert delivery by default: the
 source rank slices the stacked tensor first, then every rank broadcasts one
-expert at a time. The higher-memory full-stacked path is retained only for
-controlled performance comparisons:
+expert at a time. The higher-memory full-stacked path is a temporary
+compatibility rollback for wheels or deployments that cannot use the bounded
+split path, and it may also be used for controlled performance comparisons:
 
 ```bash
 export RTP_FASTSAFETENSORS_STACKED_MOE_MODE=full-stacked
 ```
 
-The accepted values are `per-expert` (default) and `full-stacked`. This RTP
-switch only selects how stacked MoE tensors are delivered; ordinary tensors
-continue to use the FastSafeTensors bucket and rank-local-copy settings.
+The accepted values are `per-expert` (default) and `full-stacked`; an empty
+value also selects the default. `full-stacked` adds a conservative extra shard
+to the FastSafeTensors memory preflight because it materializes a whole stacked
+tensor before RTP clones expert slices. A passive downgrade logs a warning with
+`degraded_reason`; an explicit request is reported as the selected mode. The
+additional warning is emitted only when the checkpoint actually contains raw
+stacked MoE tensors. Use `LOAD_METHOD=scratch` as the more conservative
+rollback. This transitional RTP switch only selects stacked MoE delivery.
+Bucket size, copier/backend, queue depth, producer count, loading progress and
+tensor ordering are otherwise owned by the installed FastSafeTensors
+configuration; rank-local copy-out is supplied by RTP's local checkpoint-key
+predicate.
+
+`RTP_FASTSAFETENSORS_STACKED_MOE_MODE` is a transitional, environment-only
+switch: it has no command-line flag, is not shown by `--help`, and is not part
+of the startup config dump. It is read only when the FastSafeTensors path is
+considered. Values are case-sensitive and use a hyphen; any non-empty value
+other than `per-expert` or `full-stacked` raises `ValueError` during
+FastSafeTensors selection. It has no effect for `LOAD_METHOD=scratch` or for
+weights that cannot use the FastSafeTensors path.

docs/release/breaking-changes.md

@@ -4,6 +4,41 @@
 
 ---
 
+## FastSafeTensors configuration ownership moves to AutoLoader
+
+**Introduced in:** [PR #1354](https://github.com/alibaba/rtp-llm/pull/1354).
+
+**Summary:** RTP no longer constructs a private `ParallelLoader` variant or
+forces its loader tuning through constructor keywords. Backend selection,
+buffer size, shared-memory copier, producer/queue depth, tensor ordering and
+loading progress are read from the installed FastSafeTensors configuration.
+RTP supplies only rank-local key selection and the transitional stacked-MoE
+delivery policy.
+
+**Impact and migration:** Previous RTP code forced a 2 GiB read buffer,
+selected shm versus nogds from `FASTSAFETENSORS_NOGDS`, and enabled the loading
+progress bar. New deployments should express equivalent intent through
+`FASTSAFETENSORS_CONFIG_JSON`; the exact keys are versioned upstream. For the
+current pinned wheel, a minimal nogds/progress example is:
+
+```json
+{"loader":"base","base":{"copier_type":"nogds"},"parallel":{"use_tqdm_on_load":true}}
+```
+
+At startup, degraded-but-usable FastSafeTensors paths report
+`requested_mode`, `effective_mode` and `degraded_reason`. Scratch fallbacks use
+the `falls back to scratch` marker. Missing optional APIs independently disable
+bounded per-expert delivery or rank-local copy-out; a missing package,
+AutoLoader/import failure, unmet AUTO prerequisite or insufficient memory
+preflight selects scratch.
+
+**Rollback:** Set `LOAD_METHOD=scratch` for the conservative native loader, or
+temporarily set `RTP_FASTSAFETENSORS_STACKED_MOE_MODE=full-stacked` when the
+installed AutoLoader lacks bounded dim-0 splitting and sufficient GPU memory is
+available.
+
+---
+
 ## JIT cache unified local root and remote snapshot boundary
 
 **Introduced in:** [PR #1112](https://github.com/alibaba/rtp-llm/pull/1112) (JIT remote cache).

rtp_llm/model_loader/ffn_weight.py

@@ -361,6 +361,57 @@ def _get_expert_weights(self) -> List[CkptWeightInfo]:
             for idx in range(len(self.weights))
         ]
 
+    def _raw_stacked_tensor_names(self, layer_id: Optional[int]) -> List[str]:
+        """Return concrete raw stacked keys, or empty for per-expert templates."""
+
+        if not self.stacked_ckpt_keys or not self.weights:
+            return []
+        names = []
+        for ckpt_weight in self.weights:
+            # A checkpoint template that names ``expert_id`` is already a
+            # per-expert layout. Formatting it as a raw stacked key would
+            # either raise KeyError or misclassify expert 0 as a stacked tensor.
+            if "{expert_id" in ckpt_weight.name:
+                return []
+            names.append(ckpt_weight.tensor_name(layer_id))
+        return names
+
+    def _has_raw_stacked_tensors(self, tensor_source, layer_id: Optional[int]) -> bool:
+        """Return whether every raw tensor required by this atomic weight exists."""
+
+        names = self._raw_stacked_tensor_names(layer_id)
+        return bool(names) and all(tensor_source.has_tensor(name) for name in names)
+
+    def uses_stacked_expert_keys(self, database, layer_id: Optional[int]) -> bool:
+        """Return whether the checkpoint layout requires logical expert keys.
+
+        This decision must use the immutable checkpoint database. A
+        TensorCollector populated by AutoLoader contains logical expert keys
+        even when the underlying checkpoint stores raw stacked tensors.
+        """
+
+        return self._has_raw_stacked_tensors(database, layer_id)
+
+    def _has_logical_expert_tensors(
+        self,
+        tensor_source: TensorSource,
+        layer_id: Optional[int],
+        selected_experts: List[int],
+    ) -> bool:
+        """Return whether the source already contains every logical expert key."""
+
+        if not self.stacked_ckpt_keys or not selected_experts:
+            return False
+        return all(
+            tensor_source.has_tensor(
+                self._expert_key_pattern(idx).format(
+                    i=str(layer_id), expert_id=str(expert_id)
+                )
+            )
+            for idx in range(len(self.weights))
+            for expert_id in selected_experts
+        )
+
     def _build_split_config(
         self, layer_id: Optional[int], load_config: LoadConfig
     ) -> Dict[str, Tuple[str, int, Callable]]:
@@ -369,8 +420,10 @@ def _build_split_config(
         selected_experts = load_config.get_selected_experts(
             layer_id, self.config.expert_num
         )
-        for idx, ckpt_weight in enumerate(self.weights):
-            stacked_key = ckpt_weight.tensor_name(layer_id)
+        stacked_keys = self._raw_stacked_tensor_names(layer_id)
+        for idx, (ckpt_weight, stacked_key) in enumerate(
+            zip(self.weights, stacked_keys)
+        ):
             pattern = self._expert_key_pattern(idx)
             for expert_id in selected_experts:
                 per_expert_key = pattern.format(
@@ -413,23 +466,25 @@ def _load_raw_tensor(
         if pre_sharded is not None:
             return {self.name: PreShardedTensor(pre_sharded)}
 
-        if self.stacked_ckpt_keys and tensor_source.has_tensor(
-            self.weights[0].tensor_name(layer_id)
-        ):
+        selected_experts = load_config.get_selected_experts(
+            layer_id, self.config.expert_num
+        )
+        uses_stacked_keys = self.uses_stacked_expert_keys(
+            tensor_source.get_database(), layer_id
+        ) or self._has_logical_expert_tensors(tensor_source, layer_id, selected_experts)
+        source_contains_raw_stacked = uses_stacked_keys and (
+            self._has_raw_stacked_tensors(tensor_source, layer_id)
+        )
+        if source_contains_raw_stacked:
             tensor_source = StackSplitTensorSource(
                 tensor_source,
                 self._build_split_config(layer_id, load_config),
             )
-        ckpt_weights = (
-            self._get_expert_weights() if self.stacked_ckpt_keys else self.weights
-        )
+        ckpt_weights = self._get_expert_weights() if uses_stacked_keys else self.weights
 
         convert_type = (
             self.data_type if self.data_type is not None else load_config.compute_dtype
         )
-        selected_experts = load_config.get_selected_experts(
-            layer_id, self.config.expert_num
-        )
         num_experts = len(selected_experts)
         num_ckpt_weights = len(ckpt_weights)
 
@@ -509,9 +564,7 @@ def _load_pure_tp(
             return None
 
         split_dim, segments, requires_stacked, split_func = layout
-        is_stacked = self.stacked_ckpt_keys and tensor_source.has_tensor(
-            self.weights[0].tensor_name(layer_id)
-        )
+        is_stacked = self.uses_stacked_expert_keys(database, layer_id)
         if (requires_stacked and not is_stacked) or (
             self._get_split_func() is not split_func
         ):
@@ -706,8 +759,11 @@ def _load_raw_tensor_gpu_preallocate(
     def get_tensor_names(
         self, layer_id: Optional[int], load_config: LoadConfig
     ) -> set[str]:
+        has_stacked_tensor = self.uses_stacked_expert_keys(
+            load_config.database, layer_id
+        )
         ckpt_weights = (
-            self._get_expert_weights() if self.stacked_ckpt_keys else self.weights
+            self._get_expert_weights() if has_stacked_tensor else self.weights
         )
 
         names = set[str]()