fix(server): load only the LoRA adapters a model declares (#752)

* fix(server): load only the LoRA adapters a model declares

A request's options.lora and options.lora_id now have to name a LoRA that
the model's profiles declare (a lora_paths source or served name, a
runtime.lora_id or a runtime.lora). Any other value is a 400 on the
direct and queued encode, extract and score paths, and the gateway
refuses it before dispatch, as it already does for generation's
lora_adapter. The registry refuses an undeclared id for every caller.

The PEFT mixin also refuses adapter configs whose loading changes the
shared base model: an init_lora_weights other than true, false,
gaussian, eva or orthogonal, layer_replication, megatron_config, and
non-LoRA PEFT types.

A batch whose LoRA cannot be made active now fails its requests instead
of leaving them pending. LoRA loads and unloads run while no batch of
the model is in flight, and a LoRA evicted while it loaded is unloaded
once its load finishes.

* fix(server): accept only declared LoRA ids in options.lora_id

options.lora_id names a LoRA to load, so the request merge and the
gateway now accept only a declared id there, as the registry does. A
served name stays valid in options.lora, which selects a loaded LoRA.

A LoRA that is already loaded is ready without waiting for the model's
LoRA lock, which an eviction can hold while a batch finishes.

* fix(server): load a LoRA while the model's worker is stopped

A stopped worker has shut its inference executor down, so a LoRA load or
unload that waits for its dispatch lock now runs on the default executor
instead of failing.

* fix(server): fetch LoRAs outside inference and keep loaded adapters out of errors

A request-time LoRA load now downloads its files outside inference: a Hub
LoRA is fetched once, at one commit, with only adapter_config.json and
adapter_model.safetensors, and only the in-memory attach from that local
snapshot waits for the model's in-flight batch. A failed load is held
back with a doubling backoff from 5 s to 5 min, answered with a retryable
503 LORA_LOADING and Retry-After instead of being fetched again on every
request.

The adapter config and the weights file are both validated on that
snapshot: weights other than a LoRA's own parameters, and weights
shipped only as .bin, are refused, because PEFT loads any other tensor
name into the model it wraps.

A batch whose LoRA cannot be made active fails the same way on the
pre-formed path the queue and the sidecar use as on the direct path.
When the registry can load that LoRA, it starts the load and the
requests get the retryable 503; otherwise they get a 400. Selecting a
LoRA while none is loaded is refused instead of serving the base model.
No adapter's error text lists the loaded or served adapters.

A request's LoRA options are now checked against the profile it selects,
not every profile of the model, on the server and the gateway.

---------

Co-authored-by: Krisztian Gajdar <krisztian@superlinked.com>

Author: krisztian-gajdar

packages/sie_gateway/src/handlers/proxy.rs

@@ -4498,6 +4498,24 @@ async fn queue_mode_proxy(
         return response;
     }
 
+    if matches!(endpoint, "encode" | "extract" | "score") {
+        if let Some((key, value)) = undeclared_lora_option(params.options.as_ref(), || {
+            state.model_registry.get_model_info(model)
+        }) {
+            return endpoint_error_response(
+                endpoint,
+                StatusCode::BAD_REQUEST,
+                err_code::INVALID_REQUEST,
+                oai_type::INVALID_REQUEST,
+                oai_code::UNKNOWN_LORA_ADAPTER,
+                Some(key),
+                format!(
+                    "'{key}' must name a LoRA adapter that model '{display_model}' declares; {value} is not one of them."
+                ),
+            );
+        }
+    }
+
     // Generate has its own publish + result-collection path
     // (streaming chunk aggregation instead of one-shot WorkResult fan-in).
     if endpoint == "generate" {
@@ -11733,6 +11751,79 @@ pub(crate) fn validate_lora_for_profile(
     }
 }
 
+/// The caller's ``options.lora`` or ``options.lora_id`` as ``(param, value)``
+/// when it is set and names no LoRA that the selected profile
+/// (``options.profile``, else ``default``) of the model ``entry`` returns
+/// declares. ``options.lora_id`` names a LoRA to load: a
+/// ``loadtime.lora_paths`` source or a ``runtime.lora_id``. ``options.lora``
+/// selects a loaded LoRA, so a ``lora_paths`` served name and a
+/// ``runtime.lora`` also qualify. ``entry`` is called only when the options
+/// select a LoRA; a model it does not return is not checked.
+pub(crate) fn undeclared_lora_option(
+    options: Option<&Value>,
+    entry: impl FnOnce() -> Option<crate::types::model::ModelEntry>,
+) -> Option<(&'static str, Value)> {
+    let options = options?.as_object()?;
+    let requested: Vec<(&'static str, bool, &Value)> = [
+        ("options.lora", "lora", true),
+        ("options.lora_id", "lora_id", false),
+    ]
+    .into_iter()
+    .filter_map(|(param, key, served_names)| {
+        let value = options.get(key).filter(|value| !value.is_null())?;
+        Some((param, served_names, value))
+    })
+    .collect();
+    if requested.is_empty() {
+        return None;
+    }
+    let entry = entry()?;
+    let profile_name = options
+        .get("profile")
+        .and_then(Value::as_str)
+        .unwrap_or("default");
+    let adapter_options = entry
+        .profile_configs
+        .get(profile_name)
+        .and_then(|profile| profile.adapter_options.as_ref());
+    requested
+        .into_iter()
+        .find(|(_, served_names, value)| {
+            !value.as_str().is_some_and(|name| {
+                adapter_options
+                    .is_some_and(|options| profile_declares_lora(options, name, *served_names))
+            })
+        })
+        .map(|(param, _, value)| (param, value.clone()))
+}
+
+fn profile_declares_lora(adapter_options: &Value, name: &str, served_names: bool) -> bool {
+    let runtime = adapter_options.get("runtime");
+    let runtime_names = |key: &str| {
+        runtime
+            .and_then(|runtime| runtime.get(key))
+            .and_then(Value::as_str)
+            == Some(name)
+    };
+    if runtime_names("lora_id") || (served_names && runtime_names("lora")) {
+        return true;
+    }
+    match adapter_options
+        .get("loadtime")
+        .and_then(|loadtime| loadtime.get("lora_paths"))
+    {
+        Some(Value::Object(paths)) => paths.iter().any(|(served, source)| {
+            (served_names && served == name)
+                || source
+                    .as_str()
+                    .or_else(|| source.get("id").and_then(Value::as_str))
+                    == Some(name)
+        }),
+        Some(Value::Array(paths)) => paths.iter().any(|path| path.as_str() == Some(name)),
+        _ => false,
+    }
+}
+
 fn parse_model_spec(spec: &str) -> (String, String) {
     if let Some(idx) = spec.find(":/") {
         (spec[..idx].to_string(), spec[idx + 2..].to_string())
@@ -29468,6 +29559,204 @@ mod tests {
         assert!(v["choices"][0].get("delta").is_none());
     }
 
+    // ── Declared LoRA selection on encode, extract and score ───────
+
+    const LORA_MODEL: &str = "\
+sie_id: acme/lora
+hf_id: acme/lora
+tasks:
+  encode:
+    dense:
+      dim: 2
+  score: {}
+  extract: {}
+profiles:
+  default:
+    adapter_path: sie_server.adapters.bert_flash:BertFlashAdapter
+    max_batch_tokens: 4096
+  banking:
+    extends: default
+    adapter_options:
+      runtime:
+        lora_id: org/banking-lora
+  tuned:
+    extends: default
+    adapter_options:
+      runtime:
+        lora: org/tuned-lora
+  served:
+    extends: default
+    adapter_options:
+      loadtime:
+        lora_paths:
+          legal: org/legal-lora
+          pinned:
+            id: org/pinned-lora
+            revision: 0123456789abcdef0123456789abcdef01234567
+  listed:
+    extends: default
+    adapter_options:
+      loadtime:
+        lora_paths:
+          - org/listed-lora
+";
+
+    #[tokio::test]
+    async fn undeclared_lora_option_accepts_only_what_the_selected_profile_declares() {
+        let gateway = TestGateway::new(&[LORA_MODEL]).await;
+        let entry = gateway
+            .state
+            .model_registry
+            .get_model_info("acme/lora")
+            .expect("model entry");
+
+        for (profile, name, keys) in [
+            ("banking", "org/banking-lora", &["lora", "lora_id"][..]),
+            ("served", "org/legal-lora", &["lora", "lora_id"][..]),
+            ("served", "org/pinned-lora", &["lora", "lora_id"][..]),
+            ("listed", "org/listed-lora", &["lora", "lora_id"][..]),
+            ("tuned", "org/tuned-lora", &["lora"][..]),
+            ("served", "legal", &["lora"][..]),
+            ("served", "pinned", &["lora"][..]),
+        ] {
+            for key in keys {
+                let options = json!({ "profile": profile, *key: name });
+                assert_eq!(
+                    undeclared_lora_option(Some(&options), || Some(entry.clone())),
+                    None,
+                    "{options}"
+                );
+            }
+        }
+        let not_looked_up = || -> Option<crate::types::model::ModelEntry> {
+            panic!("no LoRA is selected, so the model is not looked up")
+        };
+        assert_eq!(
+            undeclared_lora_option(Some(&json!({"lora": null, "lora_id": null})), not_looked_up),
+            None
+        );
+        assert_eq!(
+            undeclared_lora_option(Some(&json!({"profile": "banking"})), not_looked_up),
+            None
+        );
+        assert_eq!(undeclared_lora_option(None, not_looked_up), None);
+        assert_eq!(
+            undeclared_lora_option(Some(&json!({"lora": "attacker/lora"})), || None),
+            None
+        );
+
+        for (options, expected) in [
+            (
+                json!({"lora_id": "attacker/lora"}),
+                ("options.lora_id", json!("attacker/lora")),
+            ),
+            (
+                json!({"lora": "/srv/any/dir"}),
+                ("options.lora", json!("/srv/any/dir")),
+            ),
+            (
+                json!({"lora_id": "org/banking-lora"}),
+                ("options.lora_id", json!("org/banking-lora")),
+            ),
+            (
+                json!({"profile": "served", "lora_id": "org/banking-lora"}),
+                ("options.lora_id", json!("org/banking-lora")),
+            ),
+            (
+                json!({"profile": "missing", "lora": "legal"}),
+                ("options.lora", json!("legal")),
+            ),
+            (
+                json!({"profile": "served", "lora_id": "legal"}),
+                ("options.lora_id", json!("legal")),
+            ),
+            (
+                json!({"profile": "tuned", "lora_id": "org/tuned-lora"}),
+                ("options.lora_id", json!("org/tuned-lora")),
+            ),
+            (json!({"lora": ""}), ("options.lora", json!(""))),
+            (json!({"lora": 42}), ("options.lora", json!(42))),
+            (
+                json!({"profile": "served", "lora": "legal", "lora_id": "attacker/lora"}),
+                ("options.lora_id", json!("attacker/lora")),
+            ),
+        ] {
+            assert_eq!(
+                undeclared_lora_option(Some(&options), || Some(entry.clone())),
+                Some(expected),
+                "{options}"
+            );
+        }
+    }
+
+    #[tokio::test]
+    async fn an_undeclared_lora_is_refused_on_encode_extract_and_score_before_dispatch() {
+        let gateway = TestGateway::new(&[LORA_MODEL]).await;
+        gateway.add_worker("local-1", LOCAL_LANE, &[]).await;
+        let requests = |options: Value| {
+            [
+                (
+                    "encode",
+                    json!({"items": [{"text": "x"}], "params": {"options": options.clone()}}),
+                ),
+                (
+                    "extract",
+                    json!({"items": [{"text": "x"}], "params": {"labels": ["a"], "options": options.clone()}}),
+                ),
+                (
+                    "score",
+                    json!({"query": {"text": "q"}, "items": [{"text": "x"}], "options": options}),
+                ),
+            ]
+        };
+
+        for (endpoint, body) in requests(json!({"lora_id": "attacker/lora"})) {
+            let response = proxy_request(
+                State(Arc::clone(&gateway.state)),
+                json_request(&format!("/v1/{endpoint}/acme/lora"), body),
+                endpoint,
+            )
+            .await;
+            assert_eq!(response.status(), StatusCode::BAD_REQUEST, "{endpoint}");
+            let body = to_bytes(response.into_body(), 64 * 1024).await.unwrap();
+            let value: Value = serde_json::from_slice(&body).unwrap();
+            assert_eq!(value["detail"]["code"], "INVALID_REQUEST", "{endpoint}");
+            assert!(
+                value["detail"]["message"]
+                    .as_str()
+                    .is_some_and(|message| message.contains("attacker/lora")),
+                "{endpoint}: {value}"
+            );
+        }
+        assert_eq!(gateway.dispatcher.dispatched(), Vec::new());
+
+        let msgpack = Request::builder()
+            .uri("/v1/encode/acme/lora")
+            .header("content-type", "application/msgpack")
+            .body(Body::from(
+                rmp_serde::to_vec_named(&json!({
+                    "items": [{"text": "x"}],
+                    "params": {"options": {"lora_id": "attacker/lora"}}
+                }))
+                .unwrap(),
+            ))
+            .unwrap();
+        let response = proxy_request(State(Arc::clone(&gateway.state)), msgpack, "encode").await;
+        assert_eq!(response.status(), StatusCode::BAD_REQUEST, "msgpack encode");
+        assert_eq!(gateway.dispatcher.dispatched(), Vec::new());
+
+        for (endpoint, body) in requests(json!({"profile": "served", "lora": "legal"})) {
+            let response = proxy_request(
+                State(Arc::clone(&gateway.state)),
+                json_request(&format!("/v1/{endpoint}/acme/lora"), body),
+                endpoint,
+            )
+            .await;
+            assert_ne!(response.status(), StatusCode::BAD_REQUEST, "{endpoint}");
+        }
+        assert_eq!(gateway.dispatcher.dispatched().len(), 3);
+    }
+
     // ── M10: per-profile LoRA adapter validation ───────────────────
     //
     // ``validate_lora_for_profile`` is the seam the chat and generate

packages/sie_server/README.md

@@ -1060,7 +1060,7 @@ auto-retries; see `packages/sie_sdk/README.md` for client-side controls.
 | `SIE_MAX_BATCH_REQUESTS` | `64` | Maximum number of items per batched inference call. |
 | `SIE_MAX_BATCH_WAIT_MS` | `15.0` | Initial value for the adaptive first-request batch timeout. At runtime the PI batching controller steers it between `SIE_ADAPTIVE_BATCHING__MIN_WAIT_MS` and `SIE_ADAPTIVE_BATCHING__MAX_WAIT_MS`, so it is a starting point rather than a fixed wait. |
 | `SIE_MAX_CONCURRENT_REQUESTS` | `512` | Per-worker queue size; admission control returns `QUEUE_FULL` above this. |
-| `SIE_MAX_LORAS_PER_MODEL` | `10` | Maximum concurrent LoRA adapters per base model. |
+| `SIE_MAX_LORAS_PER_MODEL` | `10` | Maximum concurrent LoRA adapters per base model. A request loads or selects only the adapters its selected profile declares. `options.lora` or `options.lora_id` naming any other is rejected with HTTP 400: `INVALID_INPUT` from this server, and `INVALID_REQUEST` from the gateway (code `unknown_lora_adapter` where it answers in the OpenAI error envelope). |
 | `SIE_MAX_ITEM_TEXT_BYTES` | `2097152` (2 MiB) | Most bytes of UTF-8 one encode, score, or extract item may carry in its `text` and `metadata` together. A larger item is rejected with `INVALID_INPUT` (HTTP 400) before it is tokenized. Generation prompts are not items and are not affected. |
 
 ### Compute & precision

packages/sie_server/src/sie_server/adapters/base.py

@@ -591,7 +591,26 @@ def supports_hot_lora_reload(self) -> bool:
         """
         return False
 
-    def load_lora(self, lora_path: str, revision: str | None = None) -> int:
+    def fetch_lora(self, lora_path: str, revision: str | None = None) -> str:
+        """Bring a LoRA adapter's files to one local directory, outside inference.
+
+        Hot-reload adapters (PEFT) download and validate here, so that
+        :meth:`load_lora` only attaches from local files.
+
+        Args:
+            lora_path: HuggingFace path (e.g., "org/lora-name") or local path.
+            revision: Pinned 40-hex commit SHA, or ``None`` for the default branch.
+
+        Returns:
+            The local directory to pass to :meth:`load_lora` as ``source``.
+
+        Raises:
+            NotImplementedError: If adapter doesn't support LoRA.
+        """
+        msg = f"{self.__class__.__name__} does not support LoRA (supports_lora=False)"
+        raise NotImplementedError(msg)
+
+    def load_lora(self, lora_path: str, revision: str | None = None, *, source: str | None = None) -> int:
         """Load a LoRA adapter.
 
         This is called by the LoRA manager to load a new adapter. The adapter
@@ -605,6 +624,7 @@ def load_lora(self, lora_path: str, revision: str | None = None) -> int:
             revision: Pinned 40-hex commit SHA from the config's
                 ``loadtime.lora_paths`` dict form (#2113); ``None`` resolves
                 the Hub's default branch, as before.
+            source: Local directory from :meth:`fetch_lora`; ``None`` fetches first.
 
         Returns:
             Memory usage of the loaded LoRA in bytes.

packages/sie_server/src/sie_server/adapters/lora.py

@@ -25,9 +25,16 @@ def supports_hot_lora_reload(self) -> bool:
         if the load blocks all requests (SGLang HTTP).
         """
 
-    def load_lora(self, lora_path: str, revision: str | None = None) -> int:
-        """Load a LoRA adapter (optionally at a pinned 40-hex commit SHA);
-        return its memory usage in bytes.
+    def fetch_lora(self, lora_path: str, revision: str | None = None) -> str:
+        """Bring a LoRA adapter's files to one local directory and return it.
+
+        Hot-reload adapters do their network and disk I/O here, outside inference.
+        """
+
+    def load_lora(self, lora_path: str, revision: str | None = None, *, source: str | None = None) -> int:
+        """Load a LoRA adapter (optionally at a pinned 40-hex commit SHA), from
+        ``source`` when :meth:`fetch_lora` already brought it local; return its
+        memory usage in bytes.
         """
 
     def unload_lora(self, lora_name: str) -> None: